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INTRODUCTION 



OVERVIEW 

This report is the final report of a twelve-month study perforrQec 
by Berkeley Planning Associates (BPA) for the U.S. Department of Educa- 
tion/Office of Special Education and Rehabilitation Services (OSERS) to 
develop performance measures for supported employment programs. The 
term "supported employment" refers to a broad family of local programs 
and state-level projects designed to establish or expand supported 
employment opportunities for individuals with severe disabilities. The 
supported employment concept includes the provision of ongoing publicly- 
funded support services that participants need in order to sustain 
employment (hence the name "supported employtient"). 

As defined by the five-year National Supported Employment 
Demonstration program implemented by OSERS in September 1985, "supported 
employment" consists of the creation or expansion of compensated, 
meaningful work opportunities for individuals with severe disabilities.^ 
These work opportunities are to be created in job settings where dis- 
abled workers have opportunities for social interaction with nondisabled 
coworkers or the general public, and are intended to offer a significant 
number of hours of compensated work each veek« The National Supported 
Employment Demonstration provides states vith additional financial 
resources to be used for "system transformation" — i»e. to transform 
existing day activity and work activity programs into systems that 
create opportunities for individuals to realize their potential for 
productive work in an integrated setting. In many states, supported 
employment activities are also increasing in response to local 
initiative. Projects are expanding options within local service systems 
with or without formal state plans for system transformation, and with 
or without federal incentive funding. 

It has been the intent of this BPA study to help clarify supported 
employment program objectives and identify performance measures that can 
be used at a variety of different levels — federal, state, and local — 
to document the practices and achievements of supported employment 
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programs as they continue to receive greater attention ard are provided 
with a greater share of the program resources throughout the nation. 

The activities carried out under the BPA study were framed as 
activities to "seek consensus" across a large number of actors involved 
in a broad range of supported eniployment activities around the country. 
One of the reasons for the emphasis on seeking consensus is that 
supported employment represents a loose envelope of closely-related and 
like-minded program efforts, rather than a single program with a clearly 
defined identity supported by a single legislative mandate, funding 
source, or organizational setting. In such an environment, it is likely 
that any state efforts to generate summary data on program 
accomplishments will be the result of voluntary cooperation rather than 
hierarchically imposed data collection or reporting requirements. 

The second reason for emphasizing consensus Is the need to arrive 
at a core group of objectives and performance measures that adequately 
describe the conmon elements and intentions of supported employment 
programs that vary widely in their details of operation. At the heart 
of tills study has been a recognition of, and a respect for, the 
diversity of supported employment approaches — ranging from individvial 
placement uodels to mobile crews to group work stations (enclaves) 
within industry — as veil as the diversity of program participants, 
funding sources, and agency roles. Thus, rather than being viewed as a 
way to increase program uniformity, the performance measures discussed 
in this report are attempts to identify and reflect movement toward 
common goals, objectives and desired outcomes by a very diverse universe 
of supported employment projects. 

SUMMARY OF STUDY METHODOLOGY 

The study methodology used was primarily the gathering and 
synthesis of a large body of existing information and expert opinion 
about the goals of supported employment and ways of measuring their 
achievement. In conducting the technical work on this study, the BPA 
study team: 
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reviewed the growing body of literature on supported 
employment and related program models and topics; 
conducted telephone iscussions with a broad range of 
policy makers, program implement ors, and researchers 
involved with supported employment and related efforts, 
including key individuals from the initial ten states 
participating in the National Supported Employment 
Demonstration; 

formed an Advisory Panel of individuals representing 
various projects, universities, states, and organizations 
involved with supported employment, which prCA^ided infor- 
mation, expertise and feedback throughout the study, on 
both an individual and a group basis (with a small group 
of consultants also participating as particular sources 
of guidance and input); 

participated ii. meetings of the ten first-round 
demonstration states to discuss performance? measurement, 
data collection, and definitional issues of particular 
concern to the National Supported Employment 
Demonstration; 

gathered examples of data collection instruments, 
evaluation systems and statistical reports currently 
being used in supported employment programs and related 
fields and assessed the current availability of existing 
measures to address key aspects of program performance; 
developed a set of proposed dats. items and potential 
performance measures based on the identified program 
ob-jectives, performance questions tc ' addressed, and 
fpflsibilitx issues; and 

convened both a California jtate Forum on Supported 
Employment Goals and Performance Measures (with local 
matching funds from the San Francisco Foundation) and a 
National Supported Employment Consensus Seminar of those 
involved in the emerging supported employment to react to 
the proposed performance measures. In addition, on 
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local, state and national levels the groups discussed 
(1) the desirability and feasibility of attempting to 
implement consistent data collection across projects and 
states; (2) ideas for dissemination of proj ec t findings ; 
and (3) prioritizing research questions and areas of 
future study in measurement development and evaluating 
aspects of performance not included in the projects' 
ongoing monitoring eforts. 

Four prior reports were prepared at various stages of the study 
which reflected these research and analysis activities, including: 

• Task 2 Report; Documentation of Program Objectives, 

Berkeley Planning Associates, October 1985. This first 
report documented supported employment program objectives 
from the cl ient, proj ect, and system level perspectives; 

• Task 3 Report; Supported Employment Logic Model , 
Berkeley Planning Associates, December 1985. The second 
report described the supported employment program logic 
model by linking program objectives to the program 
designs, implementation strategies, and intended outcomes 
associated with their achievement. This report also 
began to identify data items associated with federal, 
state, and local perspectives of the 1 ogic model ; 

• Tasks A and 5: Availability of Existing Measures and 
Need for Additional Measures to Address Program Objec — 
tives, Berkeley Planning Associates, April 19P6. The 
third report s u tn in r i ■ ' nr , J. ; ? xl ab il i ty of 

.w*-ares usefu. for uocumt jg the achievements of sup- 
ported employment projects and began to assess the feasi- 
bility of expanding data collection practices to include 
additional relevant measures ; and 

• Task 6: Recommended Measures and Implementation Plan , 
Berkeley Planning Associates, May 1986. The purpose of 
this report was to identify from within the wide range of 
possible measures, those data items and performance 
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measures that address the key questions of case managers* 
project administrators, consumers, parents, caregivers, 
and state and federal observers. These recommended 
measures formed basis i - further discussion and for 

th us building ac tivities. Implementation issues 

V ere al so addressed. 

CONTENTS OF THIS REPORT 

This report traces BPA's exploration of the extent of consensus 
about the essential performance questions to be addressed in assessing 
supported emp] oyment efforts. In Part I of this report, we review the 
extent of consensus about the conceptual framework on which performance 
measurement for supported employment must be based, including 
(1) consensus about program objectives, (2) the identification of the 
intended program strategies to bring about the desired outcomes; 
(3) consensus about what performance questions should be asked (and 
answered), and (4) consensus about how to identify the areas of program 
management, operation, and outcomes for which performance measures need 
to be developed. 

Part II of this report addresses the development of consensus on a 
more practical level — the development of consensus about what 
data Items might be collected and reporte*' on a day-- co-day t by 
local pTo%^^*' ii> ar :r c t: <ins^ uiation projects in orf' r to adoaress 

^nuc oiicei-ns identified in Part I. This secor.u part of the 
describes how implementation of a core set of consensus measures 
might proceed at the project and state levels. 

Part III of the Final Report addresses the identification of future 
research priorities to address important performance questions for which 
ongoing day-to-day data collection and analysis at the project or si"3te 
level is either insufficient or unnecessary. It also discusses the 
need for further refinement of potential measures in some areas where 
there is already consensus at the conceptual level about the impc ^nce 
of an outcome domain (e.g. sociel integration, quality of life), but for 
which concrete data items have not yet been devel oped. 
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NOTES 

^For the more detailed current federal definition of supported 
employment, developed by OSERS for designing the National Demonstration 
projects, see Appendix D of this report* 
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PART I: 



ESTABLISHING CONSENSUS ABOUT PROGRAM OBJECTIVES 
AND THE ROLE OF PERFORMANCE MEASUREMENT 
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I. ESTABLISHING CONSENSUS ABOUT SUPPORTED EMPLOYMENT PROGRAM OBJECTIVES 
OVERVIEW 

After exploring the degree of agreement about supported employment 
objectives with representatives of a variety of program models and 
system level perspectives (e.g., individual project, local system, state 
system, and federal perspectives) as well as researchers on and 
proponents of the supported employment concept, we were able to identify 
a remarkable degree of consensus regarding six broad areas of desired 
program performance : 

• Meaningful Work; 

• Compensation; 

• Ongoing Support; 

• Worksite Integration and Community Participation; 

• Quality of Life; and 

• Community Change.^ 

(There were two other potential performance domains where less consensus 
was apparent. These include system transformation objectives and target 
population objectives and are discussed further at the end cf this 
chapte'^. ) 

Each of these goal areas, or domains, represents a common goal or set of 
program objectives of supported employment. However, not surprisingly, 
the degree of consensus is not absolute. Within each goal area, there 
are competing perspectives about what priority an objective should be 
assigned relative to other program objectives, and exactly how to 
approach measurement of the domain. Furthermore, each desired 
performance domain can be expressed: (1) from the perspective of an 
individual supported employment participant ( as a client objective ); (2) 
from the perspective of a local supported employment program 
( as a project objective ); and (3) from the perspective of a local, 
state, or federal service system ( as a system obj ec tive ). 

Table I summarizes the objectives for which there appears to be 
consensus in ea':h of the six performance domains, expressed first as a 
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Emplo)in ent 



Individunl lev*'! 



To Mxittiie tht oppc lity l ; dj 
roll productive vorK 

To incrt-iiac job ction by 

•iehi?ving. the foilowiii|^: 

— intpreeiing work 
appropriate level of ikilU 

— upward oobilit) potential 

— plcApant work environment 
■• lighting 

- accessibility 
~ epace 

— job security 

To maxitnire productivity 



Project level 



To provide a variety of opportvinitiei 
for ret.l work 



To fflBxinire ho'jra of available work 

To tnatch clicnti' ekilla and 
intereflta with type of work 



To provide work opportunities 
that have job security 

To aiaxicize range of job opportuoiciit 

To fltaximize productivity 



Syjte m_level 



To aaximize nujaber of individuals entering 
iupported efflploymeut ag opfiosed to day 
activity centao 

To naxiiaize hours that participants do real 
work 

To majcinizc productivity tind net contribu- 
tion Lu the "tJociai ptOuiiLl" oi tOUl 
value produced by the Gociety 



To tDstitnize range of job opportunities 



ConptnaatioD I To increase i'lcone over time 



To obtain medical and fringe 
benefits frotn employer 



To earn wages coisparable to 
productivity 



Qngoinf^ 
Support 



To receive appropriate support from 
aupported employment personnel to 
facilitate the learning of job and 
general eupUyment skills 

To receive appropriate follow-up 
services from supported employment 



To receive funds or program services 
from one or multiple agencies that 
will facilitate continuity of 
employment 



To receive appropriate aassiitance 
from nondisabled coworkers and/or 
supervisors as needed in assigned 
work activities 



To maximize opportunities to earn 
commenflurate wages 

To maximize opportunities for jobs 
with comprehenaive benefit packages 

To provide potential for increased 
earnings 



To increase the numbor of dieflblod 
individuals employed at comiaenQ urate 
wages 

To reduce dependence on transfer pByacnts 
and maximize tax contrihul ions 



To provide training in specific work 
skilli related to job placement 
as well as training in general 
employment skills 

To provide follov-up services ai 
neeued over an unspecified time period 



To provide or arrange for the pro* 
vision of ongoing training or support 
services to the disabled dien*;, thus 
assuring continuity of employment 
Over time 



To provide in-service training to co- 
workers and supervisors, sensititLRg 
then to specific (client-related) 
learning and social needs 



To coordinate the employment activitiee oi 
multiple agencies pro^/iding services to 
eeverely disabled individuals in order to 
maximize system-wide effectiveness 

To coordinate thi fiscal activities of 
multiple agencies that typically fund 
employment training and support to 
Beverly disabled individuala in order to 
maximiie ay stem-wide efficiency 

To develop coordinated techanisiiB for 
ensuring the long- tea provision of 
employment training and support (either 
through progru funding or individual 
funding flchemii) 

To increase the supply of trained 
supported employment service providers 
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Table 1 (continued) 





Individual level 


Proiect Level 


System Level 




To receive appropriate support froB 
nondiaablcd covorker* when functioning 
in the ^rkplace aocial cnvirontnent 


To provide support to covorVeri and 
super fl in dealing with disabled 
clienti needs 


To maximize the participation (fiscal 
incenavea and/or public recognition) 
of the buainoiH community in provi- 
ding support » severely disabled yorVera 


Support 
(continued) 


Intej^rat'on 


To increase interaction with non- 
disaMed workers 

To have ikillfl neceiiary to make 
a variety of tneaningful contacts 
^fith nondisablod ind dirsblod 
coworkers 


To provide opportunititi for clienti 
to socially integrate with disabled 
and nondisabled coworkers 

To provide skills necesstry to make a 
variety of meaningful contacti with 

To provide employment opportunities 


To create, support, and promote proRrams 
that fit into context of locnl business 
community 

To maximiEC the number of disabled 
individuals gaining acceaa to integrated 
work environtnentn 


At Vork 


In the 

Community 


To increase knowledge, participation 
and utilization of cownunity environ- 
menta whire social interaction can 
occur 

To develop akillB necessary to make fi 
variety of contacts with nondisabled 
and disabled individuals 


Same concepts aa above (Integration - 
Work) but within the context of 
community 


Sflme concepts as above (inlc^ration - 
Work) but within the context of coiranunity 


Oufllity oE Life 


To obtain access to and privileges 
from society which are moat commonly 
associated with work (i.e.^ pleasure 
related to connunicating with others 
about work) 

To obtain accesa to 8 greater 
repertoire of social skills — thus 
allowing for greater participation in 
a wider array of social environments 

To increase self-concept 

To obtain greater participation in an 
array of activities (e.g., less Meep 
and TV watching) 

To obtain understanding of "choice" 
and to have the opportunity to choose 
in day-to-day-situations 

To obtain the opportunity to live more 
independently 


To provide training and aupport in 
activities thnt improve quality of 
life; i.e., exposure to multiple 
environments, teaching to choose 
and providing day-to-day situations 
where the client has the opportunity 
to make choices 


To develop a syBtcm of supported employment 
that will promote client options leading 
to improved quality of life 

. 
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Tabic i (continued) 





Individual Lev(?l 


Proiect Level 


System Level 


Quality of Life 


To obtain increased access to community 
resources 

To increase family or caregiver 
satisfaction by relievinj^ pressure 
of totdl care responsibility 88 self- 
care ability of client increaBes 


See previous page 


See previous page 


(continuod) 


Co«nitxChan^ 


To change attitudes of nondiaabled 
toward persons with disabilities due 
to exposure to severely disabled 
people in the workplace and in the 

conmonity 

To dedonstrate thav participatioD of 
persons with severe disabilities in 
work» leisure/recreation, independent 
living arrangetneia Q, etc* is normative 

To demonstrate the need for environ- 
mental adaptations aa a means to 
facilitate the work and social life 
of severely disabled people 


To obtain access to an array of 

enviromnentfl typically restricted to 
nondisabled prticipati^^n; i.e^ 
fsmily night swim, "Y" fitness 
activities, public ^ransportatio- 

To promote community awareness of 
skills and special ne<:dB of severely 
disabled people 

To promote the interaction of non- 
disabled people with severely 
disabled peers 

To promote community action that 
supports the work, personal and 
living needs of severely disabled 
people 

To promote the initiation and com- 
pletion of environmental adaptations 
which will increase the level of 
participation of disabled people in an 
array of work and community environ- 
ments end activities 


Same as project-level, but from the 
standpoint of developing methods and 
procedures which promote tbesc objectives 
on a systems level 
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client objectivep then as a project objective, and finally as a system 
objective. The following discussion highlights area? of agreement, as 
well as ongoing debates about nuances of interpretation and relative 
emphasis . 

In describing supported employment objectives within each area of 
project or client performance, one question that arises frequently is 
"how do you know when an individual has reached a supported employment 
outcome?" It is clear that working definitions of supported employment 
are essential, both in order to provide guidance to states and program 
operators about the outcomes to be sought with different program fund.::, 
and to clarify what universe of individuals should be described by 
supported employment performance measures. Throughout this study^ how- 
ever, we have attempted to encourage a conceptual framework that will 
track progress tow ards supported employment objectives, even for 
individuals who have not yet achieved the levels described in current 
federal administrative guidelines for supported employment demonstration 
projects (€fg». a minimum of 20 hours of paid employment per week with 
at most eight other workers with severe disabilities in the immediate 
setting). Thus, for most of the performance domains described below, 
the program objectives are framed in terms of maximizing the potential 
of each participant, rather than achieving a specific absolute level or 
standard. Th3se performance questions describe dimensions along which 
performance might vary rather than specific expections about the degree 
of achievement. 

MEANINGFUL WORK 

There is consensus that the client-level objectives of supported 
empl oy ment include : 

• maximizing the opportunity for individuals to do real 
productive work; 

• increasing j ob satisfaction by assisting participants to 
achieve and maintain employment that: 

— is interesting, 

— is appropriate to their individual skills. 
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— takos place in a pleasant work environment » 

— has potential for job mobility^ as desired by 
participant » and 

— offers job security: and 

• uiaximizing the productivity (and thereby increasing the 
compensatit^ .) of the individual. 

Expressed at the project level, the objective of achieving 
meaningful work coalesces in the notion of achieving a good j ob-match 
for participant s. This impl ies: 

• providing a variety of work opportunities to match the 
varied interests and abilities of different participants; 

• having procedures for matching job requirements and 
client skills and interests; and 

• being able to respond to participants' needs for job 
security/work variety over time. 

From the system perspective, the objective of meaningful work can 
be expressed as: 

• maximizing the nuiLber of severely disabled individuals 
entering supported employment; 

• maximizing the range of job opportunities in supported 
empl oyment ; 

• maximizing the hours of meaningful work available to 
participants; and 

• maximizing worker productivity. 

There is little controversy over the supported employment goal of 
meaningful work. This does not mean, however, that it is easy to 
translate this goal into measurable indicators of client, project, or 
system performance. In fact, as discussed later, specific measures of 
job quality and "goodness-of-f it" have not yet emerged in the supported 
employment field. One minor source of tension in this area arises over 
how to maximize the diversity of work opportunities in the early stages 
of system transformation when program resources may permit implementa- 
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tion of only a limited number of supported employment models or job 
sites. 

COMPENSATION 

The program obj ect ive of compensati on is critical to the supported 
employment concept as one of the primary means by which increased 
community integration, and improved life quality (two other program 
goals) are expected to occur. Expressed at the client level, there is 
consensus that the goals of supported employment include: 

• increasing participant income over time ; 

• maximizing the receipt of medical coverage and other 
fringe benefits, as pait of the total employment compen- 
sation package ; and 

• ensuring that compensation level s are commensurate with 
produc tivity . 

As shown in Table 1» these objectives translate fairly directly into 
project and system-level goals oriented toward maximizing compensation 
and benefit packages. Table 1 notes that a related system-level goal 
(arising out of taxpayer interest in controlling the costs of income- 
support programs) is reducing the dependence of severely disabled 
individuals on income transfer payments (l,e. SSI). 

There are several areas where the consensus on the supported 
employment goal of compensated employment is incomplete. One such area 
concerns the benefits to be gained from volunteer or uncompensated work. 
A dissenting view holds that meaningful work integrated in a 
nonsheltered worksite, even if unpaid, can offer persons with the most 
severe disabilities a significantly increased quality of life. However, 
strong counter arguments hold that it not necessary to compromise on 
the issue of compensation, since both compensation and social integra- 
tion can be achieved, even for individuals with the most severe dis- 
abilities. 

A second tension associated with the goal of maximizing 
compensation recognizes the continuing financial and emotional 
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disincentives faced by recipients of supplementary security income (SSI) 
benefits or other income suppoit programs when they face terminating 
their eligibility for SSI and associated medical coverage by earning 
over a given earnings "cutoff" level. This is a special concern in cases 
of seasonal or unstable employment given the frequent difficulty and 
delays in reestablishing eligibility for income support. Rather than 
simply assuming that increased compensation necessarily results in 
improved financial status and security for supported employment partici- 
pants, it will be necessary to carefully study this aspect of client 
impacts as a high priority program evaluation issue. 

ONGOING SUPPORT 

The provision of ongoing support represents a service goal for the 
entire community system of services available to supported employment 
participants, as well as a goal for supported employment projects them- 
selves. Expressed at the client level, the goals of providing ongoing 
support include: 

• providing each client with the appropriate support to 
facilitate the learning of general employment skills as 
well as specific job requirements; 

• providing each client with the requisite follow-along 
support to maintain performance on the job over time 
(either from project staff or other community agencies); 

• coordinating the delivery of an ongoing package of 
publicly-funded support services necessary to sustain 
employment, from one or more agencies; and 

• encouraging the provision of ongoing support from 
informal or non-publicly funded sources, as feasible and 
appropriate (e.g., from nondisabled coworkers or 
employers) . 

Expressed at the project level, the goal of providing ongoing 
support includes; 
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• developing the program capacity to directly provide 
training and follow-up support with supported employment 
project staff; and 

• arranging for and training other community agencies to 
take over the provision of ongoing support, as necessary 
to ensure continuity over time. 

Project training of other individuals including par^rnts, coworkers, 
and job supervisors — to sensitize them to the learning and social 
support needs of specific participants can also be a project objective 
relevant to the delivery of ongoing support. 

E:xpressed at the system level, the goal of providing ongoing sup- 
port can be translated into an issue of interagency coordination and 
funding provisions. Specific system goals include: 

• coorainating the activities of the multiple agencies 



• negotiating the funding support and clarifying the 
organizational responsibilities for delivery of ongoing 
support among the several agencies providing these 
services; and 

• increasing the supply of individuals trained to provide 
supported employment services. 

The provision of ongoing support is the key distinctive feature of 
supported employment programs that permits participants to obtain or 
retain jobs in integrated compensated employment settings. However, 
there are widespread variations in practice in the types of support 
provided, what funding source(s) pay for the ongoing support, and what 
agencies provide the ongoing support. Along with these variations come 
differences of opinion about the extent to which a supported employment 
project should itself be responsible for providing, monitoring, and/or 
reporting on the delivery of support over time. These differences of 
opinion become especially apparent when the range of o.igoing supports is 
interpreted broadly to include not only support directly related to per- 
formance on the job, but also support such as assistance with 



providing services to individuals with severe disabili- 
ties to increase service availability and continuity; 
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transportation, personal care during the work day, or support in life 
activities outside of work. 

WORKSITE I NTEGRATION AI^D COMMUNITY PARTICIPATION 

We found broad consensus on supported employment objectives in the 
areas of integration at work and integration in the community. Worksite 
integration is view ed as desirable for a number of reasons. First, it 
is viewed as a powerful agent for encouraging social interaction between 
disabled participants and nondisabled coworkers, which, it is hoped, 
will result in the dual benefits of improved attitudes towards persons 
with disabilities and increased normalization of the lifestyle of the 
participants themselves. Second, worksite integration is believed to 
hive a positive effect on the skills levels and job performance achieved 
b/ disabled participants. Community integration is viewed as the 
extension of these integration goals to include interactions and 
activities that occur outside the work place, with an extension of the 
same benefits to include normalization, improved community attitudes, 
and improved life quality. 

Expressed at the client and project levels, the supported employment 
goal of integration can be summarized as the intent to increase the 
opportunities for and the frequency of meaningful social integration 
between disabled and nondisabled coworkers as a result of supported 
employment. A related objective designed to increase the level of com- 
munity integration experienced by supported employment participants is 
the intent to increase participant utilization of community environments 
where social integration can occur. 

From the system perspective, the related goals of worksite and 
community integration include: 

• promoting job sites that are integrated into the fabric 
of the local business community; 

• maximizing the number of disabled individuals with jobs 
in socially integrated work environments: and 

• maximizing the number of disabled individuals who can and 
do access the broader local community environment. 
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Conceptually, there is substantial consensus about the program's 
integration goals, Hovever, as we sha? 1 see in subseq^ient chapters, 
there are a variety of different perspectives about how to measure the 
extent of integration that is occurring. These range from computing the 
ratio of disabled to nondisabled workers, to analyzing tho job require- 
ments for social interaction in completing, job tasks, to recording (on a 
sample basis) the actual frequency and content of interactions between 
disabled and nondisabled workers. 

QUALITY OF LIFE 

"Qual ity of Life" is a residual category describing all the reasons 
(other than the work itself) why supported employment participants are 
expected to be "better off" after entering supported employment than 
before. These desired client outcomes are generally perceived as 
secondary outcomes resulting from the primary program goals of 
meaningful work, compensation, and increased worksite and community 
integration. 

At the client level, they include, among other elements: 

• an improved self-concept; 

• a broadei.* repertoire of social skills, all ow i n g 
participation in a wider array of social environments; 

• an increased choice of leisure time activities, resul ting 
from the combination of increased spending money, greater 
social skills, and greater access to community 
environments; and 

• an opportunity for greater independence in living situa- 
tion, and other activities. 

Expressed at the project and system levels, the goal of improved 
quality of life involves increased emphasis on the value of client 
self-direction or choice . Thus, as Table 1 indicates, the quality of 
life objective of supported employment projects can be summarized as the 
goal of exposing participants to different environments in which they 
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have the opportunity to make choices, and assisting them in learning how 
to choose. At the system level, the goal of improving life quality can 
be summarized as the creation of a service system that will promote 
client options leading to client choice about an improved quality of 
life. 

Although there was no disagreement about the goal of improving 
quality of life, there was some disagreement across respondents about 
whether quality of life is a discrete independent objective that needs 
to be examined separately from the goals of meaning?: ul work, 
compensation, and social integration, or whether it is adequate'^y 
represented by these more concrete domains. 

COMMUNITY CHANGE 

Another goal of supported employment is to promote changes 
community environment. From the client perspective, this obj 
includes : 

• changing the attitudes of nondisabled persons towards 
persons with disabilities through increased exposure to 
severely disabled individuals in the workplace and 
community ; 

• demonstrating that persons with severe disabilities can 
live and work and play in socially integrated settings; 

• demonstrating that environment adaptations and supports 
can facilitate the work and social life of severely 
disabled people. 

From the project and system perspectives, these goals can be tran- 
slated into community education and community change agenda items: 

• promoting community aw ar eness of the skill s and special 
needs of severely disabled people ; 

• promoting the development of community resources to 
support the work, personal, and living needs of severely 
disabled people; 
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• promoting access by disabled individuals to environments 
typically restricted to nondisabled people; and 

• promoting the initiation of environmental adaptations to 
increase the 1 evel of parti cipat ion of disabled peopl e in 
community work and 1 iving environments. 



SETTING PRIORITIES FOR q.IENT SELECTION 

Two additional performance domains were explored as part of the 
activity of identifying consensus about supported employment program 
objectives: setting priorities for client selection and system transform 
m at i on * Jhile the six performance domains previously discussed are 
closely related to the establishment of performance goals for individual 
participants, client selection and system transformation more clearly 
reflect proj ect-lv*vel or system-level goals. 

Surprisingly, client selection has not received much public atten- 
tion as a project or system-level goal. Although the philosophy of 
supported employment is clear about the goal of promoting access to 
meaningful work for individuals who were previously excluded from 
employment because they were labeled "infeasible for employment," there 
is no consensus that supported employment projects should necesrarily be 
targeting services to a single group (e.g., the most severely disadvan- 
taged). The absence of any consensus about establishing priority groups 
for client selection is probably due to a combination of the prevalent 
"zero-reject" philosophy and a fear of being forced to serve those 
clients who are the most diffcult to serve. There is an intent to use 
the supported employment approach to reach severely disabled individuals 
previously excluded from employment services, and many have no desire to 
accomplish this objective by excluding other disabled individuals from 
participation in supported employment. 

In current debates on client targeting issues, one strongly held 
view states that supported employment should explicitly target partici- 
pants currently in day activity or work activity programs, rather than 
letting the program gradually filter down to these groups by initially 
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targeting sheltered workshop participants. Yet another perspective 
views the population "aging out" of special education programs as the 
highest priority population for recruitment into supported employment. 
In the short run, it appears that participant recruitment and selection 
patterns will be governed by fundin^i, source restrictions and individual 
agency preference, rather than national consensus. 

SYSTEM TRANSFORMATION 

The final program objective — systems change — is closely related 
to the particular goals of the OSER's National Supported Employment 
Demonstration project grants, awarded to ten states in the fall of 1985. 
The establishment of systems change goals for this federal demonstration 
project is based on a recognition that, in order to be implemented on a 
broad scale, supported employment requires the ability to confront and 
redefine existing state policies, procedures, funding streams, and 
program structures.^ Ultimately, the systems change goal of supported 
employment is to significantly increase the absolute volume of severely 
disabled individuals in supported employment over time, and to increase 
the ratio of resources devoted to supported employment compared to other 
public expenditures on services for severely disabled individuals over 
time, without increasing the total federal expenditures on day programs 
for severely disabled individuals. 

The more immediate system-level objectives include: 

• Addressing and redefining current policies and procedures 
which serve as roadblocks to implementation of supported 
employment ; 

• Developing cooperation and coordination between key 
system-level agencies in the form of interagency 
agreements in order to utilize services and funding to 
the utmost; 

• Providing adequate funding, technical assistance, and 
resource development in order to transform existing 
program structures and to assist in the development of 
new programming; and 
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Providing avenues for communication betveen key actors 
involved with supported employment so that support for 
and concerns with these efforts can be addressed. 
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CHAPTER I NOTES 



^For a more detailed discussion of these findings, see Berkeley 
Planning Associates, Task 2 Report; Documentation of Program Objec - 
tives , October 1985. 

2 

Additionally, national systems change is required to overcome 
existing work disincentives created by SSI program regulations^ 



er|c 



25 



II. DESCRIBING THE PRCX^RAM LOGIC 

Establishing an understanding of the policy objectives of a new 
program initiative like supported employment is a critical first step in 
developing a framework for performance monitoring and program evalua- 
tion. The performance measures will focus on measuring the extent to 
which these objectives are achieved. However, before specifying these 
measures, a necessary second step is describing the intended program 
strategy by which the desired outcomes are to be brought about. The 
specific implementation stratt^gies selected and the relative priority 
given to different program goals will influence the selection of 
specific measures of program achievement. For example, measures of 
Maximizing the number of individuals entering supported employment might 
include monitoring overtime indicators such as: 

• the ratio of supported employment participants to 
participants in traditional day or work activity programs; 
or 

• the proportion of individuals in day activity centers 
prior to the supported employment initiative who are 
currently in supported employment. 

The choice between these measures might depend on a particular state's 
strategy of placing primary emphasize on targeting the "aging out" 
special education population for supported employment versus targeting 
primarily individuals currently in day activity programs. 

An analytic tool developed identify and describe these strategies 
is called "a program logic model." A logic model simply describes the 
program inputs (which consist of policy objectives and resources), the 
program strategies and activities developed from the inputs, and the 
desired outcomes , as well as the intended causal relatio nships between 
inputs, activities, and desired outcomes. In describing the program 
logic model for supported employment, we discovered that the supported 
employment program logic varies somewhat at the federal, state, and 
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local project levels in cerms of the emphasis placed on different 
objectives. This chapter summarizes the logic models at each level. 

TH E FED E RAL L OGIC M ODEL 

The federal logic model underlying the OSERS Supported Employment 
initiative, illustrated in Figure 1, is based on federal goals that 
emphasize (1) system transformation and (2) cost containment. These 
goals are expressed through an emphasis on promoting supported employ- 
ment opportunities by r epr ogr am m ing resources already being used to 
support day services for severely disabled individuals under existing 
programs. On the one hand, the Department of Education is clearly 
committed to advoc£.ting for service system changes that will increase 
the opportunities for severely disabled individuals tow ork at meaning- 
ful jobs, to receive fair compensation for their productive activity, 
and to enjoy an improved quality of life due to increased earnings and 
increased social integration. On the other hand, the federal program 
logic is also closely linked to the goal of cost containment. This goal 
is expressed first by the absence of any additional federal funding for 
direct services under the OSERS National Supported Employment Demonstra- 
tion. The cost containment goal is also expressed in a keen interest in 
recapturing part or all of the taxpayer investment in supported employ- 
ment through a variety of taxpayer benefits resulting from program 
participation (e.g., decreased transfer payments, increased tax 
revenues, decreased utilization of other services). 

The overall strategy identified for the federal supported employ- 
ment initiative is to encourage and support state system change efforts 



• encouraging the dissemination of information about 
supported employment to all interested states; 

• providing special demonstration funding for ten states to 
undertake state transformation projects involving tne 
state agencies with responsibility for administering 
services to severely disabled individuals; 
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Figure 1 



The Su pported Epplomnt . DfPonitrition 



Policy 
ObjectivcB 



Increase the oppor- 
tunity for severely 
disabled individuali 
to work productively 

Enable leverely dii- 
ibled iodividualB to 
enjoy an inproved 
quality of life due 
to increased coopen- 
aation and social 
integration (both at 
work and outside of 
work) 

Achieve the first 
two objectives with* 
out increasing the 
total federal expen- 
diture for services 

Recapture part or 
all of social in- 
vestnent through a 
variety of increased 
social benefita 
(decreased transfer 
paypents, increased 
tax revenues) 
decreased utiliza*- 
tion of other ser- 
vices) 



Resources 



Use of $5 million in 
supported eDployment 
(SE) deoonstration 
funds to support 
state eystcito trans- 
formation efforts 
(e.g.y interagency 
coordination^ client 
identification, pro^ 
gram start-up costs, 
reporting and moni- 
toring out cones, 
measuring outcoaei 
and impacts) 

Conversion of 
existing day activ- 
ity resources at 
the state level to 
support direct sup- 
ported employment 
services 

Investment of addi- 
tional state and 
federal funds over 
time to expand SE 
efforts 



Program 
Strateiiiej 



Encourage use of 
existing stste agen 
cy expertise (e.g., 
in state VR agencies 
and DD/KK sgenciea) 
to adminiiter and 
coordinate local SE 
projects through 
offering federal 
incentive funds 
for state transfor- 
mation projects 

Encourage develop- 
ment of a diverse 
range of supported 
employment program 
designs both within 
states and across 
statea 

Permit atates some 
flexibility in the 
definition of SE 
outcomes, SE pro- 
grams, and intended 
program clients 

Enphasite to states 
the importance of SE 
as a part of a cost 
containment strategy 
(i.e., converting 
existing day activ- 
ity services into 
supported employment 
senices rather than 
increasing the total 
supply of both types 
of services) 

Evaluate and change 
rules and regulations 
which prove to be 
disincentives to 
implementation of SE 
programs 



Implementation 
Activities 



Announce demonstra- 
tion funding availa 
bility and select 10 
states' propossls 
for funding 

Encourage the 
development of con*- 
sensus among states 
for holding diverse 
program models 
accountable for 
furthering SE pro- 
gram objectives 

Provide for techni- 
cal assistance to 
SE projects an>< par- 
ticipating BtBteb 



Desired System Trans^ 
formation Outcomes 



Increased number of 
SE participants 

Increased relative 
client volume and 
resource share de- 
voted to SE vs. day 
activity programs 

Increased priority 
given to achieving 
SE outcomes in both 
demonstration and 
Qondemonstration 
states 

implementation of 
systematic reporting 
system which permits 
monitoring of 
achievement of SE 
objectives and pro- 
gram refinement by 
state and project 
level managers 

Conversion of more 
and more day 
activity capacity to 
SE capacity over 
time; evidence of 
reduced investment 
in day activity 
centers 

Development of work" 
able system for 
incorporatiag SE 
into the VR agen- 
cies* definition of 
successful outcomes 

Development of work- 
able system for 

incorporating pro- 
vision of ongoing 
support to SE par- 
ticipants into 
existing service 
system 



Desired 

Client Outcotnei 



locreaved earnings 
(both pre versus pos 
ond in comparison to 
other day services) 

Improved quality of 
life as measured by 
client and 
parents/guardian 

Improved indicators 
of social integra- 
tion as applied in 
both work and non- 
work settings 

Stability of U 
outcomes over time 

Hecreased transfer 
payments and other 
social costs 
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© encouraging the development of a diverse range of 
supported employment program models within and across 
states ; 

• encouraging states to develop consensus on supported 
employment program objectives and on how to hold programs 
accountable for furthering these objectives; and 

• attempting to amend any federal rules and regulations 
that create barriers to the implementation of supported 
employment programs . 

The federal government has undertaken three major activities to 
implement the above strategies: (1) announcing the supported employment 
demonstration programs and selecting states to receive project funding; 
(2) encouraging a consensus-building process on program objectives an-, 
performance accountability (coordinated by the BPA study); and (3) 
providing for technical assistance to the states participating in the 
Supported Employment demonstration. The desired system transformation 
outcomes at the federal level include: 

o increasing the volume of supported employment 
participants nationwide, both in absolute terms and as a 
percentage of all severely disabled individuals 
participating in publicly-funded programs; 

• causing an observable shift in the allocation of public 
resou^ :es from day activity and work activity programs to 
supported employment programs; 

• encouraging the development and implementation of 
monitoring and evaluation practices that will permit 
documentation of the achievements of supported employment 
over time; and 

• helping individual states to overcome the organizational 
problems associated with the implementation of supported 
employment such as (1) how to change state vocational 
rehab il itation agency definitions and practices to 
incorporate supported employment participants into the VR 
caseload, and (2) how to develop a funding mechanism for 
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the provision of ongoing support to supported employment 
participants. 

At the federal level, the supported employment logic model also 
refers to a number of desired outcomes for individual participants, 
including : 

• achieving increased earnings for supported employment 
participants ; 

• achieving increased social integration for participants ; 

• achieving improved quality of life for participants; and 

• assisting participants to become independent of public 
income support. 

THE STATE LOGIC MOD EL 

Not all state*^ have chosen to identify the encouragement of sup- 
ported employment as a high priority state policy objective. However, 
an increasing number of states have developed policies and strategies 
for encouraging supported employment, including not only the initial ten 
states currently receiving OSERS system transformation grant awards, but 
also additional states pursuing this program initiative using their own 
resources and approches. A typical state logic model for supported 
employment, illustrated in Figure 2, focuses on the organizational, 
fiscal, and programmatic shifts necessary to transform the service 
delivery systems and program options available to severely disabled 
individuals. 

From a typical state's perspective, the program's policy objectives 
are to increase the number of individuals in supported employment while 
maintaining an integrated coordinated system of services for severely 
disabled individuals. This system includes service planning, client 
referral, service funding, and the provision of ongoing support at the 
state and local levels. The resources available tOb and utilized by, 
different states in developing strategies to further supported employ- 
ment outcomes vary from state to state, as do the potential strategies. 
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The basic strategies and activities in the state logic model fall into 
four categories. These include: 

(1) Proj ect Funding and Supervision : 

— earmarking a special set-aside pool of federal or 
state resources for the funding of expanded 
supported employment proj ect s at the local level, 

— actively transform ing existing day program s so that 
they achieve supported employment outcomes, 

— replicating existing supported employment models 
that have proven effective, and 

— developing or sponsoring training program s to 
increase the number of individuals qualified to 
staff supported employment projects; 

(2) Dissemination of Information : 

— informing service providers about how to develop 
supported employment models, 

— educating the business community about supported 
employment concepts , 

— informing the community at large about the nature, 
purpose, and operation of supported employment, and 

— developing and implementing data collection, moni- 
t oring, and evaluation procedures to document pro- 
ject achievements in furthering program objectives; 

(3) Correcting State Regulatory Barriers to Supported Em- 
ployment : 

— changing rules and regulations that are disincen- 
tives or barriers to the implementation of supported 
employment ; 

(4) Encouraging State and Local Organiz ational Linkages : 

— developing effective case management systems, 

— developing interagency agreements about service 
responsibilities and funding roles, 

— developing state or local management teams for the 
implementation of supported employment including 
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Pol lev 
ObiectiveB 




Resources 


iDcreaie the qud- 




Federal; Special 


beri of individuali 




Education Programs, 


participating in 




Rehabilitation 


aupported employ- 




Services 


nent prograisffiing 




AdministratioDi 






Developmental 


Ensure the availa- 




Disabilities 


bility of necessary 




State; Depts* of 


supportive services 




for persona with 




Special Education, 


severe disabilities 




Developmental Dis- 






abilities, Hental 


Transfoni state 




Health, Vocational 


systems in order to 




Rehabilitation 


provide an inte- 




Local: Rehabilita 


grated service sys- 




tion (acilities, 


ten of supported 




conoiunity orgaoita- 


employment oppor^ 




tions, business 


tunities 




contributions 



Program 
StrateRiea 



Provid.' resources 
for SE programs 

Reallocate 
resources from cur- 
rent day programs 
to SE 

Target nev 
resources to SE as 
alternative to nev 
day activity 
centers 

Replicate existing 
SE models that have 
proven effective 

Support development 
ot nev service 
delivery systems 
through range of 
program models 

DisseniDate infor- 
mation necessary 
for the development 
of SE programs 

Increase supply of 
trained supported 
employment senice 
providers 

Increase participa- 
tion of business 
community in SE 
efforts 

Integrate delivery 
of multiple agency 
services into com- 
prehensive syatcm 

Evaluate and change 
rules and regula- 
tions which prove 
to be disincentives 
to implementation 
of SE programs 



Implementation 
Activities 



Fund a range of 
program models to 
meet range of needs 

Identify and imple^ 
ment standards and 
evaluation criteria 
to measure local SE 
program performance 

Divert nev clients 
from day activity 
to SE 

Increase access to 
SE for current day 
activity cliente 

Inform and educate 
community about 
nature, purpose and 
operation of SE 

Develop and imple- 
ment training pro- 
grams for 
professionals 

Develop and provide 
TA services to 
local programs 

Develop effective 
case management 
systems 

Develop agreements 
specifying agency 
service and funding 
roles 

Develop management 
team including 
staff of partici- 
pating agencies 

Provide incentives 
for business 

community's pir- 
ticipation in SE 

I 



Desired System Trans- 
formation Outcomes 



Increased proper*' 
tion of funding 
used to support SE 
vs* day program 

Increased number of 
persons with severe 
disabilities 
employed st 
commensurate vage 

Increased range of 
support to enable 
persons vith severe 
disabilities to do 
real vork 

Increased expertise 
of service 
providers 

Increased parent 
and community 
awareness 

Increased partici- 
pation of business 
community in SE 

Shifts in caseload 
mix and/or service 
mix within agencies 

Reduction of dupli- 
cation of efforts 
across agencies 



Desired 

Client Outcomes 



Increased 

productivity 

Increased hours of 
resl work 

Increased level of 

integration in SE 
vork environments 

Reduced dependence 
on transfer 
payments 
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staff of participating agencies and representation 
from the business conimunity, and 
— identify ing priority groups ror outreach and 
enrollment by supported employment projects. 

These state-level strategies and activities are intended to bring 
about some or all of the followxng desired outcomes: (1) increased 
community awareness and support for the supported e'nployment concept; 
(2) an increased number of participants in supported employment, (3) an 
increased absolute level and/or ratio of program resources devoted to 
supported employment, (A) an increased variety of supported employment 
options available to disabled individuals, (5) increased expertise among 
direct service staff, (6) increased participation by the business com- 
munity in supported employment, and (7) reduced duplication of effort 
among state and/or local agencies. In addition, the state logic model 
also refers to the desired client—level outcomes of increases in employ- 
ment, earnings, and social integration, and reductions in dependence on 
transfer payments. 

THE LOCAL PROJECT LOGIC MODEL 

While the federal logic model is most concerned with system trans- 
formation and cost containment, and the state logic model is most con- 
cerned with monitoring, training, and establishing organizational 
linkages, the local supported employment project logic model is focused 
on the delivery of services to project participants — i.e., with the 
identification and implementation of "best practices*' in service design 
and delivery and with the achievement of improvements in the various 
performance domains or participant outcome areas described in Chapter I. 

Figure 3 suminarizes the supported employment program logic from the 
local project perspective. The policy objectives at this level are 
similar to the overall program objectives described in Chapter I; i.e., 
providing severely disabled individuals with a choice of meaningful work 
opportunities whose features include fair compensation, a socially inte- 
grated work setting, the necessary support structure to maintain employ- 



40 



33 



ment, and an increased potential to gain access to social interactions 
in the community. A final policy objective is to document project 
accomplishments. For local projects, the available resources include 
public funds in the forms of demonstration grants, reimbursement for 
service contracts, private contributions (by local foundations, 
charitable organizations, parents, and/or employers), and revenues 
generated by those projects that produce goods or services for sale to 
consumers • 

As shown in Figure 3, the service strategies and activities imple- 
mented by projects are tremendously varied. However, the common themes 
include : 

• developing a range of job opportunities to fit 
participant's needs ; 

• providing the initial and ongoing support services 
necessary to achieve general work adjustment as well as 
to meet specific job skill requirements; 

• developing supported employment in a real business 
environment ; 

• providing services that maximize participant skills and 
productivity; and 

• developing the capability to measure project 
effectiveness and participant changes. 

The box entitled "Implementation Activities" in Figure 3 provides 
examples of the generic project activities that are carried out as these 
strategies are implemented. 

Finally, the desired outcomes of the local project model shown in 
Figure 3 parallel the basic supported employment objectives described in 
Chapter I: 

• increasing participant employment levels; 

• increasing participant earnings ; 

• improving the job match between participant skills and 
job r equireiment s ; 

• increasing job-site and community integration^ 
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Figure 3 

The Supp orted E mp^ncnt ProRraa l ogic Hodel - LocbI Project ^ e^rtpect 



Objectives 



Frovtdc compenfiated 
supported cioployDicnt 
Co perfions with 
severe disabilities 

Provide a choice of 
ineaningtul, produc- 
tive work to persons 
with severe 
disabilities 

Provide employment 
situations for 
persons with severe 
disabilities in an 
integrated setting 

Develop the neces- 
fwuy support 
structure to main- 
tain persons with 
severe disabilitits 
in ctnployiiient 

Increase potential 
for persons with 
severe disabilities 
to gain access to 
and privileges from 
society in order to 
improve quality of 
life 

Docuiaent the effec- 
tiveness of 
supported etDployment 
efforts 
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Resources 



State funds trans- 
ferred from shel- 
tered work{)hops and 
day activity centers 

New funds from 
federal and state 
budgets 

Employer (private 
sector) contribution 
for wages 

local funds, e.g.) 
ARC, UCP, United Hay 
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ProgrBQ 
Strategies^ 



Maximize opportuni- 
ties to earn comtnen- 
Burate wages and 
fringe benefits 

Haxitnize range of 
job opportunties 

Hatch diente' 
skills and interest 
wilh type of work 

Promote integration 
on the job and out- 
side of work 

Provide training in 
specific work skill 
related to job 
placement as well 
as work adjuGtuient 

Provide or arrange 
ongoing training or 
support servicts 
ensuring continuity 
of employment over 
time 

Provide opportuni- 
ties for in-service 
training to co- 
workers and 
employers 

Develop within 
the context of busi- 
ness envircnsient 

Fronote iaproved 
quality of life in 
the least restric- 
tive environment 



Develop a system to 
measure project 
effectiveness and 
client changes 



kp leroc n t at i on Act iviti c s 



Identify, recruit and Assess skills 
and interests of pi'oupective clients 

Identify exirting job openings with 
appropriate wages, skills require- 
mrnts, fringe bencfilD, end job 
security 

Create new jobs tailored to meet 
clients' needs (e.g., work crews, 

Etnell busineses) 

Match clients with appropriate jobs 

Facilitate job changes as eppropriate 

Provide for maxirauu! integration in 
least restrctive environment on the 
job and outside of vnrk taske 

Provide job training within the cm- 
text of job placemenf or through 
pretraining 

Provide support on individual job 
bites or group workers under a 
common sjpervisor 

Encourage employer-providr l post- 
phjcement rupervision and follow- 
up 

Develop in-service training for 
employers and coworkers 

Encourage participation in a wider 
array of social environments 

Provide day-to-day situat^onR 
where client has the opportunity 
to choose 

Provide training in independent 
living, social, and choice-making 
skills directly or by referral 
to appropriate agency 

Document client characteristics, 
outcomes, costs, and participation 
by different agencies 
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• increasing participant productivity and socialization 
skills; 

• facilitating job s t ab il i t y / mobil i ty , according to 
participant desires ; 

9 increasing participant self-direction and independence: 
and 

• increasing the number of severely disabled individuals in 
supported employment . 

In addition, th - local project logic model includes ae a desired 
outcome, the generation of ongoing data to document project effective- 
ness and to assess the effectiveness of different service practices. 

CONCLUSION 

The descriptions of the federal, state, and local project logic 
models are useful in the development of performance measures to document 
the achievements of supported employment. First, similarities across 
the three models reassure us that, in fact« the federal, state^ and 
local project perspectives on supported employment are mutually 
reinforcing: they are oriented towards achieving a consistent set of 
desired outcomes. Second, the variations from model to model remind us 
that actors at various levels are likely to phrase program objectives 
and desired outcome measures slightly differently, because of their 
distinct outlooks. These differences are based on differences in 
priorities among objectives, differences in the level of generality or 
specificity of interest in the details of project operation and partici- 
pant outcomes,, and differences in the focus on client-level or system- 
level change measures. 

In the next chapter, we will see how the different perspectives on 
program objectives and strategies lead to a variety of different ques- 
tions about supported employment outcomes that provide the framework for 
designing data collection systems to document program accomplishments 
end analyze program performance. 
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III. ASKING QUESTIONS ABOUT PERFORMANCE 

POTENTIAL USERS AND USES OF INFORMATION ABOUT SUPPORTED E^ I^LOYMENT 

The potential users of information about the accomplishments of 
supported employment include a variety of actors, including: 

© federal and state policy makers interested in whether the 

supported employment initiative is accomplishing its 

goals; 

• state and local ad m inistrators interested in whether 
system transformation is occurring and, if so, its 
related cost and service impacts; 

• progra m ^P^ ^^^^J^^ interested in tr a ck i ng project 
resources and services provided to participants, and in 
refining service designs to maximize project effective- 
ness, as well as assessing the outcomes of services for 
participants; 

• consumer groups, and individual parents end participants 
interested in expanding community-based employment op- 
tions and in choosing the best program for themselves or 
their children with disabilities; 

• employers who id ay be considering hiring a supported 
employment participant, or sponsoring a supported 
employment group worksite; and 

• program analysts interested in tracing the net impacts of 
the investment in supported employment from the parti- 
cipant and taxpayer perspectives, as well as in analyzing 
the factors influencing project ef f ect iveness. 

Each of these individuals or groups is interested in reviewing the 
accomplishments of supported employment efforts. Some users are 
primarily interested in outcomes at the local project level, while 
others are interested in local system or state system outcomes, or in 
aggregate national statistics. Each of these different information 
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users has a stake, then, in the collection and reporting of data that 
will address their concerns. In Part II of this report, we summarize 
implementation alternatives for generating the data at thv^ project and 
system levels that will address the interests of the users described 

above. In the rercoinder of this chapter^ we review some of the perfor- 
mance questions to which different audiences are seeking answers. 

TYPES OF PERFORMANCE QUESTIONS 

There are three different types of performance questions that can 
be asked about supported employment: (1) questions asking for descrip-^ 
tive information ; (2) questions asking for comparative assessments ^ and 
(3) questions asking for explanatory analysis. 

Questions asking for descriptive information are the first step in 
documenting program performance. Examples of these questions include: 

• How many participants are being served? 

• What were they doing immediat 1 y prior to entering 
supported employment? 

• What types of jobs are they obtaining? 

• What types of support are being offered on an ongoing 
basis? 

• How much do participant s earn? 

• How much social interaction between disabled individuals 
and their nondisabled coworkers occurs on supported 
employment job sites? 

The second type of question about program performance asks for an 
assessment of the descriptive information based on some external or 
internal reference point. There are several possible sources of 
reference points: (1) the previous situation of participants (prior to 
entering supported employment), (2) any stated policy objectives about 
desired program outcomes, (3) the experience of other supported 
employment projects, (4) the shifts in the performance over time of a 
given project or participant, and (5) for more extensive evaluations, 
the experience of comparison or control group that indicates what out- 
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come would have occurred in the absence of supported employment. 
Samples of these types of performance questions follovz, using the topic 
area of participant earnings as an an illustrative example: 

• Are participant earnings greater than earnings received 
by the same individual prior to entry into supported 
employment? 

• Are participant earnings consistent with stated project 
goals? 

• Does the distribution of earnings achieved by one 
supported employment project compare favorably to that 
achieved by other proj ects? 

• Do participants' earnings increase over time after an 
individueuL enters a supported employment project? 

• Does participation in supported employmet)t result in a 
net increase in earnings for the participant ever what he 
or she would have earned in the absence of supported 
employment? 

• Does participation in supported employment result in a 
net increase in disposable income for the participant? 

The third type of question about program performance — requests 
for explanatory analysis — is concerne. with observing and explaining 
the variations in the performance outcomes cf individuals end/or pro- 
jects. These variations may be based on the influence of fixed factors 
(e.g., features of the labor market environment, client characteristics) 
or they may be i\'ifluenced by program design that are susceptible to 
change or refinement in response to research findings (e.g., techniques 
for imparting job skills or social skills; types of jobs developed by 
the project). Examples of this type of question: 

• What effect does the local labor market environment have 
on the earnings levels achieved by supported employment 
participants? 

• What effect do the types of industries and occupations 
targeted by the projects have on the earnings levels 
achieved by supported employment participants? 
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• What changes in project service strategies might enable 
participants to achieve higher earnings? 

• How do earnings vary for participants with different 
types of disabilities? 

Although all three types of questions can be asked simultaneously, 
there is usually a clear progression over the development and assessment 
of any social program, from an emphasis on descriptive questions, to an 
emphasis on com parat ive and analytic questions. During the initial 
stages of program implement ation it is usually important to provide 
answers to basic questions about what services the program is providing, 
to whom, and with what results. In data collection efforts at this 
stage there is also usually a transitional start-up phase during which 
consistent definitions, data items, and data collection procedures are 
developed and refined. 

Once the initial start-up phases of program operation and data 
collection are over, then the more complicated and evaluative 
performance questions can be asked, and performance expectations can 
begin to develop based on program experience about what levels of 
performance are likely to occur under what conditions. Simultaneously, 
analyses of variations xn outcomes can be used to identify practices 
that appear to increase project effectiveness, and dissemination of 
information about "best practices" can begin to occur. 

Although individual supported employment projects have been under- 
way for a number of years in different service sites throughout the 
nation, -.upported employment is just beginning to receive high 
visibility nationally as a program alternative for severely disabled 
individuals. Thus, during the next several years, it can be expected 
that supported employment performance questions will be primarily 
oriented towards evolving a consistent basis of descriptive information 
about the program. These descriptive data will be essential in order to 
provide a foundation for subsequent assessments of program effective- 
ness. It is the goal of this report to offer a solid set of options for 
the initial descriptive phase, as well as to establish a conceptual 
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framework to guide comparative and analytic assessments of the supported 
employment strategy. 

PERFORMANCE QUESTIONS 

In summarizing the different questions asked about supported 
employment by different groups -and individuals, we have found that the 
range of performance questions fell into ten clusters. These clusters 
include questions about: 

• compensated work; 

• meaningful work; 

• ongoing support; 

• worksite integration; 

• qual ity of life; 

• participant characteri sties ; 

• systems change and community change; 

• costs; 

• environmental factors; and 

• best practices.^ 

Below, a variety of key performance questions are listed under each 
grouping. 

Questions About Compensated Work 

Figure A summarizes the key questions about compensated employment 
outcomes and briefly describes what uses might be made of answers to 
these questions by supported employment observers, administrators, and 
operators at different levels. These questions focus on four different 
aspects of employment outcomes: 

• Hew many participants are empl oyed? 

• What are their earnings levels? 

• Hew many hours do they work in a given period (week or month)? 

• How stable is their employment? 

A final set of evaluation questions includes: 
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Fif;urc k 

Performance Q^t^^ljoj^iJiidJN^jj^ ^ l lg ors; Compensated Wor k 



^ ^ IVr f 0 rill .1 IK c_ Q 11 e 8 1 i on b 

(1) Hov nuiiiy participant a obtain paid employment? 

A, How many pflrticipants achieve supported cmploynient outcomofl 
(as jcdcrally defined)? 

B, Hov many participantfl obtain any paid eniployment? 

C, What portion of participanta ore succcafifully eerved! 



(2) H(iw much do participants cam? 

A. What ifl the level of participants' earnings? 

B, What fringe bcnelita do participants receive? 

Arc earnings a substantial portion of participants' income? 

D. How do participants' earnings compare with those of co- 
workers or other employees in comparable jobs? 



(3) Hov much do participants work? 

A. How m^tiiy hours per week do participants work? 

B. How many hours do participants work overtime? 

C* What portion of participants' time in program is spent in 
productive work? 



(4) What is the duration of tmployment? 



(5) Hov do omploymrnt status and earnings change over time? 
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These questions oddress the "bottom line" of KOKram hiiccesB and 
provide a ewary of project-* and state-level perfomi.mce. This 
infornalion ia Hflrfnl for tracking otatc and project prtf;rcfiB ov^^r 
time and can be used for crosa-etate and croBs-project (impariGons. 



Earnings are an important measure of program oulCLTiies mid are of 
particular interest to state and federal policy tnakerh concerned 
with the effectiveness of supported employment as an altt-mativc to 
traditional habilitation programs. At the project level, earniii|;6 
are an indicator of individual levels of achievement. 



One concern about traditional programs has been their difficulty in 
providing an adequate amount of work for their participants. Maxi- 
mizing the amount and proportion of productive vork is highly 
valued by promoters of supported employment. Also, the federal 
definition requires a minimum of 20 hours per week to qualify as a 
supported employment outcome. Hany individuals may qualify as 
having achieved a supported employment outcome except for the 
minimum hours r.-niuirement. Tracking the full impact of supported 
employment programs as well as any potential refinements to the 
definition requires tracking the actual number of hourr> that par- 
ticipants work. 



Given the nature of the target population and the anticipated long- 
term nature of support provided, expectations for improvement over 
time may be inappropriate. However^ since the program ii new, 
collection of data on changes over time can provide policy mkers 
with critical information about program outcomes and long-term 
trends. Program operators will use this information at the client 
level to track individual progress over time, an important aspect 
of the case management function. Also, in addition to individual 
changca overtime, project-level progress can also be tracked as 
projects are more successful in their job development and job 
matching techniques. 



Employment stability is an important measure of program success at 
administrative and policy levels, especially at an agj;ro|;ate level 
where overall trends can be observed. 



Figure < (continued) 



Jerfonnance QupsUons^ 



(6) llow^ do coiploynicnt outccmcs compare with thoee of traditional 
habiliution and day activity programB? 



(?)llov docs tltc level of cirnlngs affccl employment Btability 
(duration/retention)? 



This ifl one of the questions of etate and federal policy tnakera and 
^JJ^^J^^^ ^ ^^JO^ ^ to syatra-vide 



This 18 a research question that can aflBi8t in refinnR program 
design as prograjia weight the tradeoffe between various advantages 
ot potential employment opportunitiee, 
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• How do participant outcomes change over time? 

• How do outcomes of supported employment programs compare 



with other programs? 

• What is the relationship betw een earnings and em pi oy ment 

;j » i.4 p 1 .1 I \ J ; 

Questions About Meaningful Work 

Figure 5 describes various questions about the quality of placement 
and their particular interest to federal, state and local project 
actors. Several of the questions address descriptive topics including: 

• What kind of jobs do supported employment participants 



Other questions ask: (1) How good is the participant- j ob match? 
(2) How does the quality of placement affect other supported employment 
outcomes? and (3) What is the long-range employment mobility of partici- 
pants? These questions more appropriately fall within the scope of an 
evaluation effort, than as part of a project's ongoing monitoring 
sy stem. 

Questions About Ongoing Support 

As shown in Figure 6, there are several different kinds of 
questions about ongoing support that are of interest to federal, state, 
and local — level audiences. One kind of question asks for summary 
descriptions of the role of ongoing support within a given project 
model, in order to make comparisons across dif f er ent model s as w el 1 as 
to observe how much variation exists within a given proj ect or type of 
projects. These questions include: 

• What types of ongoing support are provided to partici- 



hold? 



When they leave a job, what are the reasons for 
termination? 



Where do individuals go after termination from a sup- 
ported employment program? 



pants in supported employment? 
• Who provides the ongoing support? 
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(jucf tionG 

(1) In what kind of joba are participantB beinR placed? 


. . Ueere 

For Riate and federal observers and project adminifitratore thece 
dtucriptoifl provide a picture of overall job dcvdopment and 
placement activities, including: (1) the range of induscrieB and 
jobs; (2) the potential for labor market saturation; and (3) the 
posGibiliiy of stereotyping or limiting of participant employoent 
optiona, These descriptors also prepare federal and etate observers 
and project-level adminietrators for aflsessing h(Tw different 
pMoyment aituationB Bnd project models affect ouuomes (e.g., 
longevity of employment, earnings, stability, etc), Case managers 
could use these deacriptors to asse&a project-participant match 
prior to purchasing services. For project stsff, particularly job 
developers and job coaches, this information is useful for; (1) 
identifying current snd future* range of tRrsetfd eaployicnt 
opportunities; (2) aaeesfling jobs which sre most beneficial for 
participants; end (3) remaining responsive to the local labor 
tnurket.^ In addition, this information can be useful to employers as 
a description of a progran'a current market, 


Are programa providing appropriate kinds of work for partici- 
pants? (participant-job match) 


Case managers, project staff and consumers would use this drncriptor 
to assess whether client-job matches ate appropiiate Hnd 
sufficiently challenging for the participant, based on task-skUl 
tnatch, State and federal observers, and program ndministrators, 
would use this information to assess project performance and changes 
in client activity over time* 


(3) Why do perticipantB terminate from jobs? from SE program? 

W Where do participants go after terminating from jobs? 
from SE prograin? 


In order to mm participant flow patterns and char n in overall 
service systeniB, federal and state observers and program 
fldminiatratore would use thia descriptor to track numbers of current 
and former SE participanta. In addition, it provides a useful 
mechanism for observing the evolution and implementing the 
refinement of practices, strategies and modelt. Case managers and 
project staff would use this descriptor to: (1) track client 
movement through the system; (2) assess the most appropriate "neit 
step," and (3) refine skills necessary to assist participants. This 
information will also be useful to job developers in developing 
strategies for marketing supported employment to employers. 


(5) To what extent does the quality of the placement influence S5 
Btability/duration? 


The findings of this research question would be useful to program 
administratore for understanding the factors aspocisted with 
improving program performance. 


(6) To what extent Jo participants have the opportunity to move 
from one employnient situation to another over time? 


This is a research issue which tests the mobility potential and 
actual occurrences of job movement in SE placemotite with the 
hypothesis being that job mobility ia as desirable an option for 

workers with severe disabilities as it is for their nondisabled co- 
workers. 
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Figure 6 

Performance Q ueetio ns and Poteiit ifll Uacrfli Ongo ing Suppor t 



QueetionG 


Users ! 


1. What types of ongoing support are provided to participantc? 
(at the work place verfius outside the work place; directly 
work-related vercua broader comiDUiiity adjuattDent support) 

2. Who provides the ongoing support to participants and who pays 
for the ongoing support to participants! 


For state and federal observers of the supported emplcyment demon- 
stration, these descriptors are uceful for undersiandnig huw 
project modelB differ regarding the provicion of ongoing support. 

For project administratora, theoe descriptors are useful for under- 
standing how the provision of support varies for diflr-rent clients. 


3. How much support is being provided! 

A, What is the mean amount of ongoing support received by the 
average project participant (in time and in cost)! 

B. What are the variations in the amounts of ongoing support 
received by supported eoployment participants? 


For project models that can identify ongoing support as a separate 
program element, these descriptors prepare federal obfiervere, and 
state and project-level a^iminifltrators to Bsseee (1) the portion of 
total operating costs devoted to ongoing support; and (2) the rela- 
tion between the public costs of support for an individual client 
versus the earnings or taxpayer benefits generated by the employ- 
ment of that client* 


4, How do the aaiountB of ongoing support to a given individual 
change over time? (Is this a model that emphasizes fnding of 
support, or is it a steady state model!) 


This descriptor would be 'igeful tc case managers tracking indi- 
vidual clients' progress, efl well as to observers interested in 
understanding the featurea of a given project, lesueB related to 
client longevity will be critical for evaluating the long-term 
benefits and coas of the supported employment program. 


5. What portion and what types of work-related support received by 
supported employment participants are publicly funded (versua 
provided at no charge or purchased by the participant)? 


This is a research concern that teats the limits of supported em- 
ployment as utilizing publicly funded support, If extensive non- 
publicly funded ongoing support in revealed* there may be reasons 
to expand the definition of supported employment, or to de- 
emphasize the distinction between supported employment programs and 
transitional employment programs. 


6» How does support influence outcome? 

A. Hov does the type and amount of ongoing support appear to 
influence the job longevity, job satisfaction, life satis- 
faction, and skills levels of supported employment parti-* 
cipants? 

B. To what extent does provision of ongoing support that is not 
directly vorknelated influence the work success and work 
longevity of supported employment cllcnta? 


This research concern will enable federal observers, and nU.lz and 
project*level administrators to assess the relative importance of 
different types of ongoing support in enhancing desired project 
outcomes, 

This is also a research concern to test the current operational 

definition of support as directly work'-related support, If other 
kinds of support are found to have a dramatic effect on job 
success, there may be reasons to expand the types of ongoing 
support that are (a) off^redi and/or (b) tracked^ 
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• Who pays for the ongoing support? 

A second set of questions attempts to arrive at quantitative mea- 
sures of the intensity and costs of ongoing support. As described in 
Figure 6, these questions ask: 

• What portion of a supported employment program's total 
staff time and costs are devoted to ongoing support? 

• How do thrj ongoing costs of supported employment compare 
to the ongoing earnings (and ultimately taxpayer bene- 
fits) generated by the employment of project partici-- 
pants? 

A third set of questions identifies research issues that might 
ultimately feed back into refinements of the supported employment 
initiative but, in the short run, would probably not be part of con- 
tinuous program monitoring. These questions include: 

• How does the level of ongoing support provided to indi- 
vidual participants change over time? 

• What portion of the ongoing support received by partici- 
pants is actually publicly funded, versus provided at no 
charge by relatives, employers, or coworkers? 

• What is the impact of providing various types and level s 
of ongoing support on desired project objectives such as 
employment longevity, job satisfaction, life satisfac- 
tion, and skill levels of participants? 

Questions About Worksite Integration 

As shown in "^igure 7, there are three key questions that have been 
asked about the integration aspects of supported employment programs: 
(l)What is the extent of integration at the work site (to what extent 
are supported employment participants placed in job sites with 
nondisabled coworkers?) (2) How much social contact or interaction 
occurs between disabled and nondisabled workers? and (3) What is the 
effect of worksite integration on community incegration and other 
participant outcomes? Further exploration of the role of integration in 
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Figure 7 

Performance Questions and Potential Users: Worksite Integration 



P p> T" o Trna n n ^ Olt«c?♦-^r^T^c? 


Potential Users i 


(1) To what extent are partici- 
pants in mainstream work 
environments with nondi';abled 
coworkers ? 


For pr oj ect- level administra- 
tors and state and federal ob- 
servers, information about the 
extent of desegregation or pro- 
portional mix of disabled and 
nondisabled workers is essen- 
tial to monitoring the shift 
from more traditional segre- 
gated work and day treatment 
Bettings* 


(2) To what extent do supported 
workers interact with non- 
disabled coworkers? 


In addition to state and federal 
policy interest in integration, 
case managers and service provi- 
ders can use this information to 
monitor participants' progress. 

Xt also can be useful in CrminaT-nno 

the quality of place ments offered 
by different employers to assist 
in job matching* 


(3) How does social integration 
at the worksite affect com- 
munity integration and other 
outcomes, such as job 
satisfaction, living skills, 
etCo ? 


These research question can shed 
additional light on the value of 
social integration at the worksite 
and assist project and state 
administrators in refining their 
service approaches. 
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supported employment programs, once they have been operationalized over 
a longer period, may clarify additional performance questions to be 
addressed. 

Questions about Quality of Lif e 

While Figure 8 depicts the various questions about quality of life 
useful to federal, state, and project level audiences, the participants 
themselves will be the main focus of this mifeasurement area in terms of 
the identification and importance of changes in individual lifestyles 
and participant attitudes. Federal and state observers, along with 
program administrators, will be looking at overall trends in living 
status, involvement in community activities, and skill development as 
supported employment programs move participants from segregated 
enviro.iment s into community-based employment. Those with more direct 
participant contact will be interested in tracking these changes on an 
individual basis in order to refine program activities and identify 
areas of growth. However, questions about quality of life provide 
participants with the opportunity to give direct feedback to all levels 
of program implementors on the daily life impacts of supported employ- 
ment . 

Questions About Participant Characteristics 

There are four basic questions that have been asked about partici- 
pants' characteristics, as illustrated in Figure 9. They include: 

• What are the characteristics of participants in supported 
employment? 

• How do these characteristics relate to the stated pro- 
ject, state, or federal goals or objectives, about the 
desired groups to be reached by supported employment? 

• How do the characteristics of participants who achieve 
supported employment outcomes differ from the character- 
istics of enrollees who do not achieve supported employ- 
ment; and 

• How do speci* f ic participant characteristics (and combina- 
tions of characteristics) influence client— level outcomes 



Figure B 

Pc rforaance Qu eBtioHB and Poten tial llBere: Q uality of Life 



Pcrfo mce OucRtiona 


Potential Users 


(1) Ilov have the living arra* .cmiB of the participanta changed 
since project entry? 

(2) Haa participant use of public tranaportation changed? 

(3) How ha6 the overall community prcacnce/participation of parti- 
cipants changed over time? 


These descriptions would indicate overall trends in the independent 
living status, akills, and community involvement levels uf supported 
employment participants. Program staff and case managers would be 
particularly interested in tracking this infonrintion on nn indi- 
vidual basis over time. State and federal obeeners, rrtiearchers, 
and program administrators would use this information to indicate 
other possible outcomes and benefits of supported employtnent. 


(4) What are the changes over time in participants'; 

• utilization of other support services? 

1 family/caregiver attitudes about employment, independent 
living, and ccramunity involvement? 

• ability to make independent decisions? 

• degree of self-esteem? 
1 health status? 

• use of leisure time^ 

• expendable income? 

I living skills, behavior, mobility, cognition, job skills, 
and conusunication skills? 


Again, these descriptors can be useful both as a general picture of 
overall state and federal bliifts occurring during supported employ- 
ment programming and as an indicator of individual participant 
evolution. TheBe questions also encourage the development of a 
format for gathering information directly from participants and 
significant othera. 


(5) What are the benefits (disadvantages) of increases (decreases) 
in participants' expendable income? 


This ia a research concern vhic^ will slso be critical to other 
levels of involved audiences as benefits to supported employment 
participants are assessed. Policy makers on state and federal 
levels vill be particularly concerned with thio descriptor as the 
costs and benefits of shifts in program dollars are assessed. On en 
individual project basis, this could be useful for tracking par- 
ticipants' changes in quality of life over time. 
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Figure 9 

Perfor mance Questions and Potential Users; Participant Characteristics 



Questions 


User s 


(1) What are the characteristics 
of project enrollees? 


The preparation of descriptive pro- 
X XX c b ux pox uxcxpanc cnaraccerxs- 
tics will be of interest to all 
observers and operators of supported 
employment projects to answer ques- 
tions about what disability groups are 
being served, what previous services 
individuals had been utilizing, etc. 
Additional related issues are what 
groups are not accessing supported 
employment and how client characteris- 
tics are changing over time • 


(2) Hov do ob.served enrollee 
characteristics relate to 
program or project goals 
and objectives? 

A. Are participants 
appropriate ? 

B. Do participants reflect 
high priority target 
categories ? 


c d L. 11 xcvcx t/X gov tf X umtf u u or px O 

ject administration where formal or 
informal service goals have been es- 
tablished, an examination will be 
made of the appropriateness of the 
enrolled participants: do they, in 
fact, have severe disabilities? Are 
they, in fact, diverting partici- 
pants from work activity centers? 
etc. 


(3) How do the characteristics of 
participants who achieve and 
sustain supported employment 
differ from the characteris- 
tics of participants who do 
not achieve supported employ- 
ment ? 


In order to determine whether pro- 
ject models are equally effective 
with all types of participants, 
researchers, program operators, and 
state and federal policymakers will 
be interest ed in careful ly examin- 
ing the characteristics of the 
clients who are least successful in 
each type of project* 


(4) How do specific client 

characteristics influence 
achievement of different 
client-level outcomes (e.g. , 
employment and earnings 
levels, job quality and life 
quality) ? 


This is the issue that is critical 
before an across-the-board compari- 
son can be made of the relative ef- 
fectiveness of different projects, 
different types of projects, and 
different state programs: how should 
performance expectations be adjusted 
to take into account variations in 
the characteristics of participants 
from one project to another? 
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(e.g., employment and earnings levels, job quality, and 
life quality measures)? 

Ultimately, research may enable observers of supported employment 
programs to develop a summary measure of "expected participant difficul- 
ty" for each enrollee in supported employment that reflects the 
statistical association between various participant characteristics and 
observed outcome levels for previous client cohorts. If such a summary 
measure or index proves feasible, comparisons of performance over time 
anc across projects could be greatly simplified. 

Questions about Community Change and Systems Change 

Those particularly interested in systems change questions, as shown 
in Figure 10, will be state and federal observers, policy makers, and 
program planners. When looked st from a local perspective, these ques- 
tions also have significance to regional and community administrators 
and project staff. The questions ask: (1) What changes have occurred 
in the service delivery systen at the state and local levels? (2)Have 
the intended supported employment objectives been achieved? (3) What 
are the quantifiable results of supported employment outcomes system- 
wide? (4) What is the impact of supported employment on funding flow and 
participants flow? and finally, (5) Are current service delivery 
mechanisms adequate given the influence of supported employment program- 
ming? The latter two questions are research issues which may ultimately 
address the need for more extensive systeirs change. 

Questions About Supported Employment Costs 

Figure 11 summarizes the key questions about supported employment 
costs and briefly describes what uses would be made of these questions 
by supported employment observers, administrators, and operators at 
different levels. The first set of questions asks: how are supported 
employment projects funded and what constraints do different funding 
sources place on project operations? The second set of questions asks: 
What are the "typical" costs of providing supported employment and how 
much variation exists in providing supported employment opportunities 
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Figure 10 

^^-^-^■^'MijtjpjliyniPr' '■LtjaU!lcjjj_5y8t mi Cki^ t 



Pcrfortnance Qiicelione* 




(1) What are the varialione in intended ByBtema change for; 

1 increaBed volume of supported employment outcomps? 

1 increaBed proportion of supported employment outcomeB to the 

total eyatem? 
• incrcai]od range of joba available? 
9 overcoming barriers diid creating incentiveB? 


~ Potential Oecrs 

For federal observers, this descriptor vill be useful for under- 
stan ing overall and comparative systm tranaformation ,icrofB states 
bosed on the intended objectives of the supported m\M 
initiative. i J ' 


(2) What IB the volume of supported employment outcomes? 

A» What iB the volmc of supported csiplopcnt outcofae*) over 
time? 

B» To vhat extent is supported employment an addition to or a 
replacement for current programniing? 

C» yhat portion of the population who could benefit from dim- 
ported employment are being served? 

(3) Wiat iB the flow of money - who is paying for what and how does 
that change over time? 


Stote policy aakers and progra planners will use these descriptors 
to assess (1) hov supported employment has restructured previous 
progrsTOinj and (2) how nany of the target population has been 
served. For progran administrators, these will be key descriptors 
"1 targeting appropriate participants for appropriate smices. 


(4) What infl'jsnce has the aupported employment initiative M on 
the service delivery system over time? 

A, What are the changes in service delivery mechanismB over 
time for; 

• case managetnent procedures? 

• rate Betting structures? 
1 documentfition/fflonitoring? 

t ability to provide ongoing support? 
t referral mechanifims? 

• interagency mechanieini)? 

B. Vhat are the changes in the flow of participanta over tine? 


In order to capture the range of system change impacts on state 
service delivery mechanisne, funding streams, and participant flow 
patterns, state policy makers and state-level program planners will 
need to have specific descriptive and quantitative informction on 
the changes and/or long-term trends occurring as a result of sup- 
ported employment progruiB. More specifically, these descriptors 
will assist m short- and long-range planning efforts in states and 
across states, as well as providing a pool of coj)p,tatlve informa- 
tion on best practices. 


(5) What is the extent of systems change at the local community 
level? 

A, What changes ore occurring locally in the service deliviery 
ttjfbicn in areas sucn os referral Btructures, interagency 
coordination, and sourcee of funds to pay for oncoine sup- 
port? 

B. Are there chances in the hiring practices of local employers? 


For local progrm administratora, case managers, and project staff, 
u will be critical to assess trends in both the public sector and 
t e private sector in order to expedite service delivery and iob 
placement efforts, ^ 



Fic^rc ID (continued) 



(6) Hhal is the relalionship between changefi in funding flow and 
participAnt flow? 

A. How do the changeB in the flow of Itic iiioroy affect the flow 
of the participantfl through tlie Byatwn? 

B, Itov do the changes in the flo^ of participento affect the 
flow of the money through the Byfltem? 



(I) Ifl there a need for a centralized outreach and aaaeBBnient 
ID' hanism? 



Thu IB a research concern vhich will provide information on 
whether participants are followini; the funding nrm^ or the 
funding nxi^m are following the porticipaiits ovi-r time, 



Thi« alflo IB a research concern as federal and Btate obhcrvers and 
local project administrators look at coordination cffurtB over lime 
and assess the ciost effective ways to restructure Bcrvice delivery 
systras for persons with «)e»fere difiabilities, 
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Figure II 



miojPMclloA^LoaanOpicntiilJj^^^ 



.P-^'I f PIRi^llce, ([ye fit i 

(1) What fundini; fiourccB are being ubcJ to support the operation of 
Bupportcd ttiploymcnt projects at the lornl level (including 
revenue produced by project activitiea)/ For projcctB that have 
access to more than one funding source, how arc funds fnim 
different sources coordinated? (e.g., funding allocated by 
client, funding allocated by activity). 



(2) Vhat is the cost of one month of supported (Tnploycient for one 
participant? 

A. What is the coat of one month of supported einploynient for a 
typical ifticipant? 

B. How do monthly costs vary 

• by project? 

• by program model? 

• by client characteristics? 

• by length of tine in program? 



or fe eral obseners and state adainistrators, analynng the 
ov of funds to program operators is csBential fnr iinderBtandinR 
the fiscal constraints on the program, as well as the require- 
ments imposed by each funding source regarding clients to be 
served, or limits on the timing, scope, or objective nf services. 



(3) How are project costs allocated across different types of activi- 
ties, such as: 



general administration? 

community outreach, education, and advocacy? 

client-intake assessment, job match? 

direct client supcrvision/training/ongoing aupport? 

client specific casework and advocacy? 

generalised job development? 

purchase of outside services for clients? 

evaluation/record keeping? 

revenue-producing activities (including cost of wages 
paid for work perfonned)? 



W What are the net fiscal impacts of supported eoployment froo the 
taxpayer perspective? 

A. What are the average monthly reductions (increases) in trans- 
fer payments resulting from participation in supported employ- 
ment? 



B. 



What are the average monthly reductions (increases) in the 
cost of Social services utilized as a result of participation 
in supported employment? 
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C. Vhat are the average monthly increaiea (reductiom) in tax 
rcvcmicB paid as t result of participation in aupportcd 
enploynent? 



At all evele, fm federal policy aakera to state administrators, 
to local coordinating counciU and local supported wrlcymcnt 
projects, inforuation about the typical costs and the rnnge of 
variation in costs for different types of participants in dif- 
ferent types of supported enployment models will be used to set 
expectations for individual performance and to compare different 
projects and different aiodeU. Ultimately, an assessment of 
variations in costs will also set the state for a comparison of 
the net benefits of alternative modela. 



The primary users of this detailed analysis of project activities 
will be project Banaj?ra thtneelvea, as well as state monitors, in 
order to assess; (I) how project resources are being allocated; 
2 how these expenditure patterne are changing over ime; and' 
(3) what apparent effect the investment in different activities 
hae on reasure of project effectiveness and cost-effeciivrnesa. 



!lie questions of net fiscal impacts require extensive inveitiga- 
tions of the indirect outconea experienced by individual supported 
coployment participants, as well as tracking the experiences of 
a control or ccmparison group who did not enter supported employ- 
oent. For thia reason, they will not be of day-to-day interest 
to prograa administrators. However, program managers iind pslicy 
oakers at all levels will be vitally interested in what research 
reveals about the relationships between their direct outcome 
monitoring neasurse and net impact estimatea. Specifically, 
decisions made by program policy makers about client turnelinR 
will lean heavily on research findings about which types uf pat- 
ticipanta (1) create the greatest net taxpayer return; and (2) 
experience the greatest personal financial benefitb fr« partici- 
pation in supported eoployBent, 



Figure 11 (continued) 



Pprf onnnnce iucntione 




(5) From the participant perspoctivo, whot are the not financial 
ijipacta of Eupportcd employment? 

A. What are the incrc. -s (dccreaaeB) in uncompcneated monthly 
work expenses aasocutcd with participation in supported 
employment? 

B. What are the increases (decrcaaes) in expendable income 
associated with participation in supported employment 
(increased earnings minus taxes minus reductions in trans- 
fer payments) 


. PotonlinI llpprs 1 

See the comments under 4, above. 


W How do the Man monthly costs of supputted ttnployment compare to 
1 11 Buoii montniy costs ot other day programs which strve a similar 
client population? 


The users of these questions and answers will be any actors at 
the federal or state levels who are primarily interesli'd in sys- 
tems transforation, that ia, in reallocating existing and planned 
new investments in day services for the target population so as to 
realue the greatest taxpayer and client benefits. The first step 
IB to compare costs of alternative or complmentary services. 


(7) How do the estimated mean monthly taxpayer and participant bene- 
fits of supported employmeit compare to the mean monthly benefits 
generated by other prograns which serve a similar client popula- 
tion? ' ' 


The second step is to compare the taxpayer benefits generated by 
slternative or complementary services, 


(8) How do the estimated monthly project costs and taxpayer benefits 
vary across different supported employment projects and types of 
projects? 

(9) How do the estimated net financial benefits of supported emplcy- 
oent from the participant perspective vary across different types 
of supported employment projects? 


finally, researchers and policymakers will be interested in under- 
standing the variations in supported employment costs and benefits 
as they interact with vsrialions in participant characteristics 
and aenice approaches over time. Ultimately, research may reveal 
that certain models have a cost advantage for certain types of par- 
ticipants, 
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via different project models to different types of participants? The 
third set of questions, which will be of particular interest to project 
managers themselves, ask about the costs of the distinct activities 
funded under supported employment. 

Three final question areas concern (1) developing estimates of the 
net fiscal impacts on the taxpayer of operating supported employment 
programs; (2) developing estimates of the net financial impacts on the 
individual participants of supported employment programs; and (3) com^ 
paring the marginal costs and benefits of investing additional funds in 
supported employment versus alternative or complementary services. 



ADDITIONAL QUESTIONS 



The clusters of performance questions described above offer a range 
of performance questions which can be used at state and federal policy 
levels to monitor program performance and which can also be used by 
state and local administrators to monitor their own progress. These 
indicators also provide the basic framework for conducting a more com— 
prehensive evaluation effort. However, in addition to the eight domains 
above, there are a number of other performance questions that we recom— 
mend for inclusion in future performance evaluations. These include: 
(1) What are the practices and procedures associated with program sue— 
cess? and (2) How do external and environmental factors influence sup— 
ported employment outcomes? These performance questions are not 
presented here in the same level of detail as the eight domains above. 
The specification of data elements and specific performance measures in 
these areas are perhaps premature and would emerge during the develop— 
ment of a detailed evaluation design. The users and uses of information 
are essentially the same across these additional performance questions. 
Evaluation information will be useful to state and federal policy makers 
and state and local administrators in refining the supported employment 
seirvice delivery approach. A comprehensive understanding of what works 
and why it works may lead to revisions in the definition of supported 
employment and state and federal implementation and funclng policies. 
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Questions About Pest Pract icej^ 

One of the major objectives of any program evaluation effort is to 
produce information that can lead to improvements in program design and 
implementation. Evaluation of practices and procedures contributing to 
success woul i include such questions as: 

• How do various practices and procedures differ between 
supported employment programs and more traditional 
vocational and day activity programs? 

• How do the characteri sties of program opera t or s seem to 
affect outcomes (e.g., type of agency, history and 
experience, organizational structure and staffing)? 

• Which models and approaches seem to w ork best for 
different types of participants? 

• Which models and approaches seem to work best for 
different kinds of local conditions (e.g., urban versus 
rural, different kinds of labor markets)? 

• Are there different specific "best practices" that seem 
to work well across different models and populations 
(^•g». participant assessments, job match techniques, job 
coaching and training techniques, case management and 
follow-up procedures, funding mechanisms, interagency 
coordination) ? 

• What is the relationship between different outcomes 
(e.g., earnings, employment stability, integration) and 
different program models? 

• How do the different outcomes relate to each other? What 
are the trade-offs between maximiz ing earnings, hours of 
and degree of integration? How do these relate to levels 
of support provided and program costs? 

• What are the qualifications of program staff and what 
kinds of training do they participate in? What are the 
training needs of different actors -i the supported 
employment service system? 

• What are the roles of various actors in the supported 
employment program (e.g., state and federal policy- 
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makers, universities, consumer groups, service provider 
organizations, etc.)? How do shifts in leadership occur 
as the program matures, and how do these shifts influence 
philosophy and policies in the field? 

Questions About External Factors 

There are a variety of factors likely to influence outcomes that 
are not under the control of the supported employment program itself. 
An assessment of how these factors affect outcomes is essential to a 
comprehensive understanding of program performance^ Should any of these 
factors prove to have strong relationships with specific program 
outcomes, at some future time it might be desirable to include such 
factors in the ongoing comparative assessment of program performance. 
Such an approach might include weighting outcomes according to specific 
factors in order to take into account differences in local environments 
when comparing one program to another. 

A number of different factors have been identified to date as being 
likely to affect performance to some degree. These include: 

• work/income disincentives, such as potential loss of SSI 
and medical benefits; 

• local labor market characteristics and the availability 
of suitable jobs; 

• local economic conditions and feasibility of creating new 
jobs and business ventures; 

• general economic trends and conditions in the state or 
nation as a whole; 

• local service environments (::;ervice rich or service poor) 
and the avail ab ility of adjunct support services for 
community living and personal/social needs; and 

• cooperation from other agencies and programs and 
interagency coordination within the 'tonstraints of state 
and local bureaucratic structures, policies and 
regulations. 
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Another question sometimes asked about the external constraints on 



• Is there a danger of labor market saturation in some 
communities? How is this (might this be) addressed? To 
what extent does supported employment affect other 
groups' access to entry-level jobs? Are there industries 
or job types with untapped potential? 

The eight primary topics used to structure performance questions in 
this chapter appear in Part II of this report as a basic framework for 
presenting a set of core and supplementary performance measures which 
are suggested for implementation by states and individual projects on an 
ongoing basis. Additional topics are suggested for periodic or ad hoc 
research efforts by projects, states, or outside evaluators. 
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CHAPT ER III NOTE 

^ThiB list bears a deliberate resemblance to the list of areas of 
consensus on program objectives described in Chapter I, In their evolu- 
tion from objectives to performance questions to clusters of consensus 
data items, the following changes ir title occurred: v orksite and c om ~ 
munity integration was divided intc two topics — worksite intej^ration 
and part of the topic quality of life — and p rogram costs, environmen- 
tal factors , and best practices emerged as new categories. A clear 
trend also emerged, whereby those topics most readily lending themselves 
to quantified measurement increased in scope, and th se topics difficult 
to measure in a consistent fashion began to shrink* 

In the development of specific data items and measures to address 
these clusters of performance questions, the eight measurement domains 
underwent two additional title changes — c ompensated w ork issues are 
addressed by measures of employment outcomes , and meaningful work issues 
addressed by a somewhat expanded set of concepts about quality of em- 
ployment . 
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IV. IDENTIFYING RECOMMENDED DATA ITEMS AND hfflASURES 
INTRODUCTION 

While the first part of this report focused primarily on the 
conceptual framework within which the study evolved, the following 
section present s a practical framework for implementing common ongoing 
measures of supported employment efforts. These common measures fall 
within the eight measurement domains discussed previously: 

• employment outcomes; 

• quality of employment; 

• opportunities for worksite integration; 
0 ongoing support ; 

• quality of life; 

• participant characteristics; 

• systems change; and 

• supported employment costs. 

Within each domain, the BPA study team used the wide range of 
information gathered, through the literature and discussions with 
informed respondents, to identify which data items and measures might 
best be considered as: 

• iL2££» universal measures that provide information on 
an ongoing basis, including data necessary for ongoing 
program monitoring and for summarizing performance at 
state and federal levels. These are also measures that 
would be most useful if implemented across projects 
and/or states; 

• supplementary , or additional measures that need not be 
the same across states and projects. These may include 
particular areas that a project would want to monitor for 
itself on an ongoing basis, as well as measures that 
reflect local and/or project model pecularit ies; and 

• evaluation , or periodic measures or special studies that 
would be conducted for a representative sample. 
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occasionally, or only after the program has demonstrated 
longevity. 

Using these criteria, the study team presented a potential set of 
core, supplementary, and evaluation measures to 35 individuals from 
around the country for discussion and consensus building during a day 
and a half seminar. These individuals represented federal agencies, 
state and local public and private agencies from th*^ fields of develop- 
mental disabilities, vocational rehabilitation, ard special education, 
as well as educational institutions and research centers. The National 
Consensus Seminar occurred on May 20-21, 1986 in Washington, DC, and was 
structured to (1) generate response to the potential measures; 
(2) create an opportunity for a diverse group of involved persons to 
explore the extent of theoretical agreement and consistency in practice 
on program goals, implementation practices, and measures of program 
accomplishment: (3) allow federal, state, and local representatives to 
exchange information to develop an understanding of the dimensions of 
supported employment as it emerges around the country; and (4) establish 
communication links between program operators, researchers, and policy 
makers. During the seminar, consensus-building focused largely on 
individual proposed data items rather than on the aggregate performance 
measures that might be constructed from them. The consensus-building 
process centered on those measures considered by the group to be core or 
universal . 

This chapter reflects the group's work during this consensus- 
bui'Lding process through tables displaying the set of core consensus 
measures and a set of supplementary or additional measures for each 
domain. In addition, each section attempts to briefly address: 

• Why is this domain important to measure/what do we want 
to know? 

• What are the constraints in measuring this domain? and 

• What are the implications for collecting data in terms of 
the consistency and timing of data to be collected? 

The lists of core consensus measures represent a set of data items 
and measures that would ideally be implemented by all supported employ- 
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ment programs in a consistant way. While these consensus measures do 
not mean to imply a federal intention to mandate standardized data 
collection, they do represent a minimal set of agreed-upon measures and 
a statement from those active in the field about what the supported 
tiuiployment program might ho\d itself accountable for and/or v;hat 
describes the program's intent. In addition, areas were identified 
where further research will be needed to develop measures and 
mcthodol ogies. 

The supplementary sets include measures that would entail choices 
made at proj ect and/ or state discretion about what particular inf orma- 
tion would be useful to collect. The specific measures included are 
suggestions or representations of ways to expand the core set of ongoing 
measures. Other measures which reflect project or community-specific 
concerns could certainly be added to the list of possibilities. Special 
studies or evaluation topics for each domain are addressed in Part III 
of this report, along with overall re:^earch issues in supported employ- 
mc^nt programs. Chapter V of this section discusses the various concerns 
which arise when attempting to develop a data collection strategy and 
implementation plan for ongoing performance monitoring. 

f 

EMPLOYMENT 0UTCX»1ES 

The key feature of supported employment that distinguishes it from 
day activity programs is paid employment. Employment exists when an 
individual's activities create goods and services that have economic 
value, and when he or she receives payment for work from an employer or 
customer. Therefore, the first step in measuring performance and 
perhaps the most important measure of success of supported employment 
programs is assessing the extent to which employment outcomes are 
achieved. These outcomes can be organized into four general dimensions: 
(1) actual placement/employment; (2) stability of that employment; 
(3) earnings; and (A) hours worked. These areas were identified as the 
key aspects of employment outcomes during the study, and are reflected 
in the core consensus data items and recommended measures listed in 
Figure 12A. 
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Figure 12A 
BMPLOYMENT OUTCOMES 
Cove Consensus Data Items and Reccmnended Measures 



Conecnsue Data Items 


RecoBunended CoreHeaGuree 1 


• Enrolled in S£ program 


• total # of participants served (active enrolleee) during j 
reporting period | 



• Obtained paid employment 



il and % of participants in paid 
during reporting period) 



nployment (at a- time 



Achieved SE outcome (current OSEKS 
definition) 



• ^ and X ot participants who huve achieved supported 
Gaployment outcomee (current OSERS definition) 

• X of participants in paid employment who have achieved 
supported employment (current OSERS definition) 



• Hourly wage at placement 



• mean hourly wage at placement 



• Hourly wage at end of reporting period 



• mean hourly supported employment wage (mean wage for all 
participants who have achieved supported employment out- 
comes) 

• mean hourly wage for all participants in paid productive 
work (including participants who have not [yet] achieved 
supported empl oyment outcomes ) 

• mean monthly earnings for all participants in paid pro- 
ductive work (including participants who have not [yet] 
achieved supported empl oyment outcomes) 

• % of employed participants/participants achieving sup- 
ported employment earning minimum wage or above 



• Received fringe benefits: health 
coverage 

— yes 

— no 



• mean o.onthly supported employment earnings (mean monthly 
earnings for all participants who have achieved supported 
empl oyment out c omes ) 

• total earnings across all participants who have achieved 
supported employment outcomes for reporting period 

• total earnings across ail participants (including those 
who have not yet achieved supported employment outcomes) 
for reporting pe riod 



• Total hours worked during reporting 
period 



• mean hours worked per week in supported employment 



• if weeks employed since enrollment, for 
participants with SE outcomes 



• mean weeks employed since enrollment for current par- 
ticipants with SE outcomes 



0 weeks employed during reporting 
period, for participants with SE 
outcomes 



• mean 0 weeks employed during reporting period for current 
participants with SE outcomes 



• 0 weeks employed with present employer, 
for participants with SE outcomes 



• mean length of time {0 weeks) with present employer for t 
participants with SE outcome ■ 
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Measuring these dimensions will enable us to answer the questions: 
To what extent is the participant in supported employment involved in 
paid productive work, how paid and h^w productive is the work, and is 
that work retained over a period of time? In addition, this information 
will be an integral part of an internal assessment of project 
performance as well as providing bench marks against which to compare 
outcomes across projects. These comparisons will facilitate the 
identification of: (1) differences among supported employment projects; 

(2) differences between supported employment and other programs; 

(3) "best practices," or exemplary methods for achieving best outcomes; 
and (4) a long range picture of the supported employment program as it 
develops. 

As evidenced in Figure 12A, the issue of defining who is a 
participant in supported employment is central to the consistent 
measurement of employment outcomes. The National Consensus effort 
resulted in a determination to collect data on all supported employment 
participants, defined as active enrollees . with certain measures 
designed specifically to capture outcomes for those who fall within or 
achieve the current OSERS supported employm<-it guidelines (i.e., working 
at least 20 hours per week, in a group of eight or fewer disabled 
workers, and receiving publicly-supported, on-the--job support). Thus, 
the consensus items reflect a decision about questions of definition and 
priorities — decisions which will be necessary to make on a broader 
scale to promote the greatest possible consistency in data collection. 
Other National Consensus decisions which are evident in the core data 
items are: 

• an agreement to measure both hourly wage (to capture 
individual client level progress and changes in produc- 
tivity) and total monthly earnings over the reporting 
period — collected at the client level on a monthly 
basis and reported either monthly or quarterly on an 
aggregate level; and 

• an agreement to collect both total hours and number of 
weeks worked during the reporting period for participants 
with SE outcomes (as defined by OSERS). 
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Measures of weeks worked for participants who have not (yet) 
achieved supported employment (as defined by OSERS) were determined to 
hot supplementary (see Figure 12B). In addition, measures that looked at 
chr.nges in earnings and employment status over time (by quarters) were 
aJ so considered supplementary, as were measurer of fr^Lnge benefits other 
than health coverage and more detailed information about hourR worked 
per week. It is interesting to note the consensus reached about 
measuring hours and weeks worked in addition to earnings measures. 
Participation in work on a ,egular and sustained basis is valued as a 
goal in and of itself, independent of the wages and production yielded 
by that activity. 

The potential for collecting consistent employment outcome data 
across states and projects seems high, given an effort to utilize common 
definitions. Projects and states must also consider implementation 
issues. Much of this information can be collected at the client level 
relatively easily, and then aggregated at the project and state level. 
It may be somewhat more difficult to collect client level data on fringe 
benefits, but measuring these at the employer level might not accurately 
represent whether or not participants actually receive fringe benefits. 
Ultimately, it is critical to look at all employment outcomes of 
supported employment in light of its goal for "normalized" employment. 
This will mean considering the data in the context of nondisabled 
coworkers and/or the standards of that particular industry. 

QUALITY OF KMPLOYMKKT 

One of the goals of the supported employment initiative is to offer 
participants an opportunity to shift from the o-Jten monotonous tasks 
performed in sheltered employment to productive and meaningful work 
involving variety in tasks, the opportunity to acquire new skills, 
increased job satisfaction and security, and employment mobility. Some 
programs have goals that even go further than placing clients in "real" 
jobs in the community and have given priority to finding "non^ 
traditional" placements for individuals with severe disabilities, as 
alternatives to what some consider to e stereotypical placements. 

o 
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Figure 125 
EMPLOYMENT OUTCOi<ES 
Additional or Supplementary Data Itesu end Other Poeaible Heasuree 



Additional or Supplementary 
Data Items 



Other Possible Measures 



Received fringe benefits; sick leave, 
vacation time, other 
— yes 



Z cf employed participants receiving: sick leave, vaca- 
tion leavei other fringe benefits 



Value of frinee benefits as a percen- 
tage of earnings 



• mean fringe benefit rate (value of fringe benefits as per-! 
centage of earnings) 



Mean hours worked per week during 
period employed 



# of supported employment participants working: 

— 1-10 hours per week 

— 11-20 hours per week 

— 21-30 hours per week 

— 31-40 hours per week 



Total hours spent in day program or 
work activity other than supported 
employment during reporting period 



X of total day program hourt spent in supported employ- 
ment 



p weeks employed since enrollment 
(for participants who have not [yet] 
achieved supported employment out- 
comes ) 



mean weeks employed since enrollment (for current par- 
ticipants who have not [yet] achieved SE outcomes^ 



f weeks employed during repo'^ting 
period (for participants who have 
not [yet] achieved SE outcomes) 



mean # weeks euployed during reporting period (for par- 
ticipants who have not [yet] achieved SE outcomes 



# weeks employed with present employer 
(for participants who have not [yet] 
achieved SE outcomes) 



mean length of time (# weeks) with present employer (for 
participants who have nor [yet] achieved SE outcomes) 



Z weeks employed since firet placement 
(for all participants) 



mean Z weeks employed since placement (for all partici- 
pants) 



Duration of program participation 
— # quarters in program 



mear # quarters in program for current participants 
mean # quarters in program for terminees 



toployment status at end of 2nd quarter, 
4th quarter, 8th quarter, 12th quarter, 
etc., after enrollment (based on con- 
sensus data items) 



number and Z of total program participants employed at 
end of 2nd. 4th, 12th quarter after enrollment (a par- 
ticipant is included in this measure if the reporting 
period is his or her 2nd, 4th, 8th, 12th quarter after 
enrollment) 



Earnings during 2nd, 4th, 8th, 12th 
quarter after enrollment (based on 
core data items) 



mean monthly earnings during 4th, 8th, or 12th quarter 
after enrollment (c. participant is included in this 
measure if the reporting period is his or her 4th, 8th, 
or 12th quarter after enrollment) 
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Thus, in order to measure the performance of supported employment 
efforts, the first step is to collect information about the nature of 
the work itself. 

In addition to the employment outcome meaFures described 
previously, most of the existing measurement systems reviewed and 
analyzed by the BPA study team included some information about the 
nature and adequacy of the work placement. While this was often 
informal and descriptive in nature, a number of potentially useful 
examples of more formal procedures for collecting and reporting data 
about work placement characteristics exist. Much more work is needed, 
however, to develop measures that can address the full range of impor- 
tant questions about the quality of the employment situation including: 

• What is the nature o f the work — is it an improvement 
over "make-^work" in sheltered settings? Is it real work? 
Is it meaningful work? Ic there a good match between the 
worker and the job tasks? Is stereotyping occurring? 

• What is the quality of the work environment — are appro- 
priate job accommodations being made? What is the 
quality of the physical environment — is there enough 
light and air? Wha: is the quality of the interactions 
between workers and supervisors — are there i ppropriate 
avenues for communication and exchange of information? 

• Why do supported employment participants leave their jobs 
and w here do they go after termination — are supported 
workers leaving job situations for reasons similar to 
other workers? Where could improved supported employment 
program practices enhance job stability and retention? 
What are the external barriers that prevent supported 
workers from retaining employment? khive do participants 
go after termination from supported eiaployment? 

• How is supported employment interacting with the labor 
m arket — what kinds of jobs are beinp; accessed by 
supported workers? What kinds of labor markets developed 
on behalf of supported employment projects are being 
tapped successfully? Unsuccessfully? Is supported 
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employment filling existing jobs, creating new jobs, or 
both, and to what extent? 

While not intended to comprise a comprehensive assessment of 
quality, several measurement areas describing basic employment 
characteristics were included by the National Consensus Seminar in the 
core measure set (see Figure 13A). These measurement areas include: 
type of employer, type of job, and reason for termination trom the 
supported employment project, and were viewed by seminary participants 
as descriptive or proxy measures rather than measures of quality or 
program performance accountability. While the basic set of descriptive 
data gathered from these core measures will provide some of the 
information necessary to address the performance questions above, the 
current state of the art in defining and measuring quality of employment 
has certain gaps. For example, measuring the appropriateness of the job 
match is difficult. Current measures reviewed for this study are being 
used during the job placement process itself as a part of job task 
analyses and skill assessments during service delivery. However, these 
processes do not lend themselves to evaluating the ongoing quality of 
the match or the goodnes s— of — f i t between the worker and the job over 
time. Further development of measures in this area are needed, so they 
can be included in a set of core measures. In addition, any measurement 
of the quality of the work environment itself requires the development 
of new measures. 

Another factor integral to measuring quality of employment involves 
relating the particular supported employment jobs to the opportunities 
in the labor market. While a comprehensive picture of the types of jobs 
performed by supported workers would be useful on national, site and 
local levels for policy makers and proj ect implement ors alike, caution 
must be used in com paring the range of jobs developed jy individual 
projects given the wide variation in local settings. These measures 
will also be useful descriptors in analyzing reasons for termination and 
participant destination status after program termination. 

Figure 13B provides examples of expanded response categories for 
each of the categories of reasons for termination, which have been 
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Figure 13A 

QUALITY OF EMPLOYMENT 
(Empl oyment Characteristics) 
Core Consensus Data Items aod Recooiaended Measures 



CcnsencuG Data I tec: 


Recotiicendoc Core Measures 


• 


Type of employer 


• 


and 2 of 


participants employed by different types of 








empl oy ers 




private nonprofit 










— private for profit - small 










business or large corporation 










government - local, state or 










— federal 








• 


Type of job (develop standardized 


• 


# and % of 


participants in different types of jobs 




categories, referencing D.O.T., 








S.O.C. codes, etc.) 








• 


Who pays wages? 


• 


it and % of 


participants receiving wages from each source 




— SE program 










— employer 










— other 










Reason for termination by category 


• 


and % of 


participants terminating for Primary reason by 




(primary and secondary reason) 




ca tegory 




— employee performance reasons 










(involuntary departure) 


• 


and % of 


participants terminating for secondary reason 




— emp- /er reasons other than 




by category 




employee performance (involun- 










tary departure) 










— employee reasons (voluntary 










departure) 










— provider reasons 










— other 








• 


Participant destination/status after 


• 


# and % of 


terminating participants leaving supported 




termination from supported employment 




employment 


for each type of destination 




program, for example: 










— school 










— wait list 










— no service 










— institution 










— another SE program 










— day ac t ivity program 










— sheltered workshop 










— independent competitive employment 










— retired 










— other 










— unknown 
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Figure 13B 
QUALITY OF EMFLOYMEOT 
Additional or Supplementary Data Items and Other Possible Measures 



Additional or Supplementary Data Items 



• Type of industry (develcp standardi red 
categories, referencing D. 0. T. , S,O.C., 
etc. ) 



Other Possible Measurgs 



• and X of participontc er.;^lcyed by type of industry 



® Reason for termination by specific types 
(more than one may be indicated) 



A. Employee performance reasons 

(involuntary departure), for example 

— attendance 
— — compliance 

— emotional outbursts 

— personal hygiene 
independence 

— on task (off task) 

— quality of work 

— responding to instructions 

— social skills 

— speed 

— task completion 



B. Employer reasons other than 
employee performance (involun- 
tary departure), for example: 

— change in job duties 

— laid-off, facility closed 

— laid-off, cutback in staff 

— laid-off, seasonal 

— replaced by another worker 



C« Employee reasons (voluntary 
departure), for example: 

— quit, due to change in 
relationship status (e.g,, 
married, divorced, etc, ) 

— quit» due to pregnancy and/or 
parenting responsibilities 

— quit, for better job 

— quit, didn't like job 

— quit, health reasons 

— leave of absence 

— mcTved from area 



D» Provider reasons, examples to be 
Gevel oped 



E. Other reasons, for example: 

— quit, due to parental caregiver 
pressure 

— quit, due to financial aid inter- 
ference 

— quit, due to transportation 
problems 

— deceased 

— other 



• # and X of participants terminating for each type of 
employee reason (involuntary departure) 



f and X of participants terminating for each type of 
employer reason (involuntary departure) 



# and Z of participants terminating for employee reasons 
(voluntary departure) 



* and Z of participants departing for provider reasons 



• 0 and Z of 



participants terminating for other 
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grouped accorcing to e^aplcyer, employee, or provider reasons. As 
supplementary data, this measure permits recording more than one reason 
for job termination^ Fi^>. <Rxample, an employee's behavior which results 
in an involuntary termination may be the worker's only way of expressing 
dislike for the job. Thus, two expl anat ions, and possibly more, exist 
for that termination. The more detailed response categories would 
permit recording all the reasons for termination that apply to each 
situation. 

It would appear, then, that data on termination reasons may be 
collected at the client level, while other quality of em pi oy ment 
measures may be appropriate to project level data collection and 
measures to demonstrate overall trends in job placement and movement 
within the context of local communities and projects. Research efforts 
needed to refine measures of job quality include not only studies of new 
approaches to measuring quality and job match, but also studies to add 
depth to the ongoing descriptive data, such as: (1) job mobility 
comparisons between those in supported employment and other workers; 
(2) studies of how supported employment projects are (or are not) 
accessing noncraditional jobs within the labor market; and (3) st^-^xee 
of how to use participant self-reports to define and describe the 
quality of employment and changes over time. The current core set of 
measures should be viewed as initial attempts to develop valid 
descriptive measures of the nature and adeqixacy of jobs. The importance 
of expanding current expertise in measuring employment quality will need 
to be reflected in future research efforts as the supported employment 
program evolves. 



OPPORTONITIES FOR WORKSITE INTEGRATION 



The integration of individuals with severe disabilities into the 
workplace is a key element in the supported employment efforw. Worksite 
integration focuses on providing opportunit iets for disabled and non~ 
disabled co-workers to interact in a variety of settings and situations 
including the immediate work environment, lunchroom, break times, or 
during travel to and from work. 
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The results of this study have indicated the importance of going 
beyond measures that simply indicate whether individuals with tevere 
disabilities are being placed into jobs where there are nondisabled co- 
workers present, A more complete definition of integration needs to 
irrlude the existence and extent of meaningful contact between disabled 
and nondisabled coworkers. Evidence indicates that this type of inte- 
raction provides a teaching and reinforcement mechanism for social and 
work skill development. It is also an indicator of normalized working 
conditions and the opportunity to establish sccial relationships between 
people who have formerly been isolated from one another. 

While integration at the worksite is the primary integration 
objective of supported employment programs^ study results have also 
stressed the multidimensional nature of integration and the importance 
of integration outside of work. For many participants, success at work 
will be dependent in part on participation and success in integrated 
non-work environments. Persons with disabilities interact with non- 
disabled individuals when buying clothes to wear to work, when buying 
food, and while riding public transportation, as well as during partici- 
pation in recreation/leisure activities. For many disabled individuals, 
community integration is not only an essential pr e-requisite to Buc- 
cessful integration at the worksite, but is also an outcome of suc- 
cessful participation in supported employment. (See also Quality of 
Life .) 

Community integration is a concern that has been given a great deal 
of attention in recent years by administrators, service providers, 
researchers and policy makers in the fields of independent living, 
developmental services and special education. It is an area that is 
relatively new to vocational rehabilitation and employment policy. 
Perhaps because it is considered by many to be a secondary outcome in 
the context of employment programs, measuring community integration is 
not yet well-developed or well-defined in the field. Thus, few concrete 
examples exist of measures of the performance of supported employment 
programs in furthering community integration objectives. Further 
development would be needed to construct simple summary measures that 
could be included in the evaluation of supported employment impacts. It 
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appears unlikely that community integration measures would be included 
in ongoing performance assessment, at least in the initial stages of 
program development. Thus, the proposed core measures listed in Figure 
lAA concentrates on opportunities for integration in the worksite only. 



to define and collect information on the extent and nature of contacts 



disabilities to nondisabled individuals is somewhat easier and the data 
are relatively easy to collect. However* study results indicate that 
developing indicators of the degree and kinds of interaction would 
result in measures that are more reflective of the accomplishment of 
integration objectives than measures of the numbers of nondisabled 
workers at the job site. 

Some attempts have been made to develop client-level measures of 
integration which include features of both work setting and community 
interaction. While still in the initial stages of implementation and 
reliability testing, such attempts bode well for the development of such 
measures . 

Until such developments occur, or until adaptations are made of 
existing integration measures from other related fields, current 
measurement of integration focuses on measures of proportion, proxy 
measures, and/ or qualitative evaluation efforts. As Figure 14A 
indicates, the core consensus measures include basic quantitative 
measures of the numbers of supported workers in the group support 
structure, such as work crew or a work station, and the presence or 
absence of nondisabled workers (other than staff) in the immediate work 
setting. In addition, the seminar participants proposed an additional 
measure describing the supervisory structure utilized in the supported 
employment model which includes distinctions between mobile versus sta- 
tionary and individual versus group structures. Each type of super- 
visory model has different implications for degree of opportunity for 
integration, such as increased opportunities in individual/stationary 
structure*:; for contact and interaction between disabled and nondisabled 
workers. Thus a description of the structure would provide valuable 
insights about the possibilities for worksite integration. This is 



A major constraint in measuring integration is the question of how 



between individual workers. Monitoring the proportion of persons with 
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Figure 14A 
WORKSITE nWEGRATION 
Core Consensus Data Iteais and Recommended Measures 



Consensus Data Items 


Recommended Core Measures 


• Type of supervisory structure: 

— mobile versus stationary 

— individual versus group 

• Presence of nondisabled workers 
(other than staff) in immediate 

work setting during work day: 

— yes 

— no 

• Number of supported workers in 
group support structure (group 
work station or work crew) 


•* 

• # and % of supported workers 
in each type of supervisory 
structure 

• % of employed participants 

who work in a setting with 
nondisabled workers present 

• % of program participants in 
group support structures of 
eight or less supported 
workers 

• % of program participants in 
group support structures of 
six or less supported workers 

• % of program participants in 
group support structures of 
four or less supported workers 
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true, also, in the case of measuring the number of supported workers in 
the group support structure — another core consensus measure* All of 
those measures, however, were viewed by seminar participants as proxy 
measures until future research efforts can address these measurement 
issues further. 

Another proxy measure was developed during the Consensus Semirar 
and was included as a supplementary measure (see Figure 14B). Tracking 
whether contacts are required betveen the supported vorker and super^ 
visors, coworkers and customers was viewed as one way to begin to study 
interactions, although this still does not answer the question of how to 
measure the extent and nature of those contacts. 

Determining the proposed supplemental and core data items measuring 
integration raised a number of other issues or constraints. The 
questions were raised: Is integration a practice or an outcome? A 
guideline or a measure? A part of quality of life within and outside of 
the workplace? Although no conclusions were reached, the group was able 
to determine "zhat the current sets of proxy integration measures woula 
be most appropriately collected at the project level, as they are not 
meant to be measures of individual participant success or failure. 
However, as client level measures of social interaction begin to emerge 
as supported employment programs mature, further discussion is needed to 
develop a common underst an^^ing of the role of integration measures in 
measuring program performance. 



ONGOING SUPPORT 
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The provision of ongoing support is the "mainstay" of the supported 
employment concept and that aspect which most distinguishes supported 
employment from other employment outcomes. The primary objective of 
ongoing support is to enable an individual with severe disabilities to 
sustain employment at a level that ha or she would be otherwise unable 
to achieve. 

The most common concept of support is the prep a supported 

employment program staff person to provide on-the- job training and 
assistance. While concept s about the nature of these responsibilities 
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Figure 14B 

WORKSITE INTEGRATION 
Additional or Supplementary Data Items and Other Possible Measures 



Additional or Supplementary 
Data Items 


Other Possible Measures 


• % of disabled workers in work 
setting 

— total # of workers in worlv 
setting 

— total # of workers with dis- 
abilities in work setting 

• During the course of the work 
day» is the supported worker 
required to interact with the 
following: 

— supervisors, 

— coworkers, 

— customers? 


• if and % of supported workers 
in work settings where less 
than 10% of the workers have 
disabilities 

• if and % of supported workers 
in work settings where 11-50% 
of the workers have disabili- 
ties 

• if and % of supported workers 
in work settings where more 
than 50% of the workers have 
disabilities 

• % of supported workers 
required to interact with: 

supervisors 

coworkers 

customers 
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vary, most models of supported employment include on-the-job support in 
three major areas: training, supervision, and behavior management. In 
addition to specific work-related support, the concept of supported 
employment may also include support that is indirectly related to the 
employment objectives such as transportation assistance, "peer support," 
e.nd services to employers. Other types of support indirectly related to 
employment are those related to life activities outside of work such as 
assistance with independent living and social skills development. There 
is substantial difference of opinion about the role of supported employ- 
ment programs in addressing needs outside of the workplace. 

Diversity also exists in the intensity and duration of the support 
provided to participants. While the basic premise of ongoing support is 
that it is provided on a potentially permanent basis, some programs 
operate models in distinct phases in which on-the-job training is faded 
out over time and replaced with follow— along services that focus more on 
independent living, social skills, and general problem solving than on 
the work itself. In other programs, the structure of the job-related 
support is ongoing by its very nature, as in group work station or 
mobile crew models. 

Although the concept of ongoing support is key to understanding how 
supported employment differs from other forms of employment, discussions 
about the implementation of performance measures related to ongoing 
support have been inconclusive. It was suggested that measures of 
ongoing support should be used as descriptive measures of different 
supported employment models, rather than as normative measures , since it 
is not the ca se that pr ov id ing one 1 ev el or ty pe of support is be 1 1 er or 
worse than another. The ideal is to provide the necessary support to 
assist each participant to work as independently as possible, which will 
be a "ifferent mix for each participant and may vary for a given parti- 
cipant over time. 

As described in Figure ISA, three data items were identified as 
core measures for the domain of ongoing support. These include: 

• a description of the types of support which the project 
generally provides to its participants (e.g., assistance 
in transportation to and from work or self-care, training 
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Figure 15A 



ONGOING SUPPORT 





Core Consensus Data Items 


aud 


Recommended Measures 


Consensus Data Items 


Recommended Core Measures 


• 


lype of support provided: 


• 


type of supp:>rt provided by 








proj ect during reporting 




— transportation 




period 




— self care 








— job skills 








— social interaction/behavior 








— community living skills 








— other 






• 


Whether support is provided: 








— at the work place 








— ouLside the work place 








— both 






• 


Type of support structure: 


• 


# ar.d % of participants in 








different types of support 




— individual setting. 




structures 




— group work station 








— mobile crew 


• 


approximate prcport:-on of 




— small business run by 




supported employment par- 




or for employees with 




ticipants receiving each 




disabili ties 




type of support during 




— other 




reporting period 
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in job skills, social interaction/behavior, or community 
living skills, or other support); 

• whether the proj ect generally provides support at the 
vork place, outside the vork place, or both; and 

• a description of the general framework within which 
support is provided by the project (i.e., in an 
individual placement set ting, in a group w ork station, in 
;i mobile work crew, or in a small business run by or for 
employees with disabilities) . 

These data items can be used to develop proj ect— level measures that 
describe the features of particular supported employment projects. 
Discussions with representatives of a variety of projects revealed that 
not all projects document variations in the types or amounts of support 
received by individual participants for use in a client- level measure. 
However, it would be useful in describing each supported employment 
project to know the approxiaiate proportion of participants who received 
each type of support during a given reporting period, in addition to 
knowing whether the project has the ability to provide support of each 
kind. 

Figure 15B describes several additional data items and measures 
that co^lcl be used to describe the delivery of ongoing support in more 
derails * . ese data items may be perceived as relevant to sdue supported 
employment models and not others, and may be available at j^e projects 
and not others. The data items include: 

• what funding 6ource(s) are used to prov ide ongoing 
support ; 

• what organizations or individuals actually provide 
ongoing support (e.g., supported employment st r.f f versus 
employers or coworkers, versus another local service 
agency) ; 

• hov many hours of support are provided each month, in 

total, and by type of support ; and 

• what the monthly cost of providing ongoing support is, in 

total and by type of support. 
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Figure 15B 
ONGOING SUPPORT 

Additional or Supplesentary Data IteMS and Ot:her Possible Measures 



Additional or Supplementary 
Data Items 


Other Pos sible Measures 


• 


Funding source (s) for publicly- 
funded ongoing support (i.e., 
agency and budget line) 


• 


Funding sources utilized 




Provider(s) of publ icly --funded 
ongoing support (i.e., supported 
employment staff versus compensa- 
tion to empl oyer, coworker, or 
local service agency) 


• 


Provn der s util ized 


• 


Total monthly hours of support 
per participant 


• 


Mean hours of ongoing 
support per participant 
during reporting period 


• 


Total monthly hours of support 
across all participants for each 
type of support 




— by type of support 

— in total 


• 


Total monthly costs of support 
per participant 






• 

• 


Total monthly costs of support 
across all participant s for each 
type of support 

Total earnings by supported 
employment participant s 
during the reporting period 


• 


Mean and distribution of 
ratio of cost of ongoing 
support to < arnings gener- 
ated per participant during 
reporting period 



See Data Items under Employment Outcomes 
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These data items can be used to compute descriptive statistics 
including the aean hours and dollars of ongoing support provided each 
month per participant. By using data on mean monthly earnings 
(described under Employment Measures) an additional summary measure can 
be computed of the nean monthly cost of providing ongoing support 
compared to the meazi monthly earnings generated per participant during 
the reporting period. 

In summary^ vhile the descriptions of ongoing support are not 
intended^ by themselves, to provide an assessment of program success, 
they are critical for understanding variations in the service strategies 
and resulting costs of different models of supported employment. The 
institutional linkages and funding flows by vhich continuity of 
services/ongoing support is maintained over time is another key 
implementation issue with wide variation. A careful study of this 
aspect of supported employment may yield ideas about "best practices" 
that will prove valuable in replication efforts. 

QUALITY OF LIFE 

Improved quality of life for individuals with severe disabilities 
has consistently been raised as a desired outcome of supported employ- 
ment. How improved quality of life is defined and measured varies from 
person to person, but in general the concept is viewed as increasing 
those activities which bring pleasure and self-worth to the severely 
disabled individual over his/her lifetime^ The activity of work by its 
very nature has intrinsic values which can bring a sense o: satisfaction 
and fulfillment, and can increase access to and pr±\ ±.±ti^^z> from society 
which are most commonly associated with work (i.e., making friends, 
sharing resources, and expanding social skills and environments). 

Parallel to the personal intrinsic gains obtained by an individual 
with severe disabilities when he/she has the opportunity to work are the 
positive changes thct can occur for the family or primary caregiver of 
the employee in supported employment. Reduced family stress, and 
freeing up time for family merr.bers are additional quality of life out- 
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comes. Quality of life can also improve as a result of the expanded 
activities an individual can oarticipate in or the additional items 
he/she can obtain because of his/her increased purchasing power and/or 
greater access to choices. In addition, improved quality of life can 
result irom a combination of more self-confidence, self-direction and 
increased exposure to a greater array of options in the world. 

It is only reasonable that quality of life issues focus on the 
individual and how the individual makes choices to obtain greater satis- 
f notion. Yet, in the area of enpported employment, the project and the 
overall system provide the structure which determines whether the 
individual with severe disabilities understands choice, has had practice 
making choices, and knows how to choose for the purpose of satisfying 
his/her needs. The concept of individual choices is considered to be an 
important aspect of quality of life. 

Through discussions with respondents and reviews of the current 
a-^ailable literature on measure development, the BPA study team identi- 
fied the following aspects of quality of life as areas for potential 
measurement : 

• type of living arrengenent at program entry and over 
time ; 

• use of nainstreaM transportation at program entry and 
over time; 

• degree of coaonunlty integration and participation; 

• changes in Quality of Life Indicators, e.g., health 
status, self-direction and opportunity for choice, 
attitudes of family, participants and/ or caregivers, 
self-esteem, skills levels, etc.; and 

i^xpendable income of participants, rograir. entry and 

time. 

During the National Consensus Seminar participants indicated that 
the above areas did not adequately define the concept of quality , 
suggesting that these aspects actually described "lifestyle 
characteristics" rather than "quality of life." Some participants 
raised the issue that using the term "quality" implied a value judgment, 
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whereas looking at changes in characteristics over time would not 
involve defining the standards for quality. Participant self-report was 
suggested as a way to capture this information without arbitrary values 
being placed on changes by non-participants. No specific data items, 
however, received group approval for inclusion in the core data set, 
although the domain itself was considered important to continue to 
pursue through research and special studies. 

Several constraints emerge when considering the "how-to's" of 
measuring quality of life. At this point in time, a refined and 
reliable quality of life questionnaire and response index, which could 
be used across supported employment participants, family members and 
caret,i\ers, does not exist. This kind of instrument would enable base- 
line client-level data to be collected at program entry and then 
compared over time with results from administering the same 
questionnaire at regular intervals. Some work is currently being done 
in this area, and there appears to be growing interest in further 
research efforts. There is a concern, however, that a standardized 
questionnaire may not be feasible, given differences in local 
communities and services. Limitations in what are considered 
"acceptable" evaluation methods, i.e., use of participant observation 
techniques or self report, may also curtail the development and use of 
innovative creative approaches to quality of life measurement, to the 
extent that policy makers and funding agencies may not consider them to 
be valid. For non-verbal participants, however, using alternative forms 
of self-report may be the most effective method. 

The measurement of quality of life changes also involves the issue 
of vhether defining or r^t^n<^^rr 'z'. t-^' ac;; ^s. choice and autonomy , 

po^i>_ --tr or even dec:>iraDxe. W most sem inar participants agreed 

bat these concepts were integral parts of quality of life, the question 
of definition and ultimately, of feasibility of measurement arose. When 
considering choice, there appear to be three major dimensions: (1) the 
opportunity for choice, i.e«, whether an individual is living, working 
and playing in an environment where the possibility of self —direction 
exists; (2) the ex t ent of choice, i.e., how m uch possibility for 
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decision-making exists; and (3) the kind of choice, i.e., the range of 
areas available within which to make decisions. 

While no consensus was reached on core measurerb, as shown 
Figure 16, several measure -ere it. uded as supplemental proxies for 



arrangement, use of mainstream transportation, and a self-report measure 
of degree of independence in living situation. It was noted that 
measuring type of living arrangement should include lateral movement 
within the category as potentially more or less independent than the 
participant's previous situation. Measuring use of mainstream transport 
tation would also be a supplementary measure used at project discretion, 
given that transportation and community mobility is locality specific. 
The self-report measure was added during the seuinar to allow for 
participant definition of independence. 

Assessing quality of life changes for persons with disabilities 
remains a critical issue for development and refinement of currt-nt 
measures. The intent of supported employment is to positively influence 
quality of life through meaningful paid work in integrated settings. 
Whether or not it ever becomes the responsibility of programs to track 
these influences remains to be seen, but as an avenue for encouraging 
the active sel f ^assessment of the impact of supported employment on 
participants' lives, quality of life measurement represents an es?: 
part of evaluation efforts. 

^Ah^ r^iKACTii.RISTICS 

The target population for supported employment programs is that 
group of individuals traditionally excluded from vocational 
rehabilitation services due to the severity of their disabilities. 
Supported employment is designed as an alternative to day activity ^nd 
habilitation progams. Most proponents of supported employment 
subscribe to the concept of the "zero reject model", which suggests that 
no individuals would be turned away from supported employment pre ms 
solely on the basis of limited abilities. This does not imply "total 
inclusion" or that all individuals must work. However, the "zero 
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Figure 16 

QUALITY OF LIFE 
(Lifestyle Characteristics) 
Supplementary Data Items and Other Possible Measures 



Supplementary 
Data Items 



Type of 1 iv ing arrangement at 
program ent ry and over time, 
for example: 

— with parents or relatives 

— in group home or resident ial 
f acil ity 

— in semi-independent living 
situation 

— in state hospital 

— other 



Other Possible Measures 



# and % of participants in 
each type of living arrange- 
ment at the end of each 
reporting period 



# and % of participants 
changing to more independent 

settings during reporting 
period (sel f- report ) 



Self-report of degree of 
independence within living 
setting 



il and % of participants re- 
porting increased indepen- 
dence within living situation 



Use of mainstream transporta- 
tion at program entry and 
over time: 

— does not utilize mainstream 
transporta t ion 

ut il izes mainstream trans- 
portation with assistance/ 
prompt s 

— utilizes mainstream trans- 
portation independently 



# and % utilizing mainstream 
transportation with assistance 



and % utilizing mainstream 
transportation independently 



# and % of participant s 
increasing use of mainstream 
transportation during 
reporting period 
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reject" concept does imply that individuale will not be denied the 
opportunity to do prod- ctive work just because their capacity for work 
is less than the standards set for individuals without disabilities. 

Because of the interest in using supported employment to reach out 
to individuals excluded from work opportunities in the past, and because 
the supported employment approach may be appropriate for a vide range of 
individuals with physical, mental, or emotional disabilities, it will be 
important for programs to collect data which capture the range of 
participant characteristics. Three major questions arise when assessing 
the performance of supported employment programs: 

(1) Who is being served : 

(2) Who is being successfully served? 

(3) How do outcomes and service approaches vary for 



In the past, the client characteristics recorded on supported 
employment intake/assessment records at the proj ecc level have usually 
included variables that are viewed as relevant by program operators in 
the development of a service plan to meet each individual's needs, as 
well as data that are readily available at the time a participant enters 
the program. Given the potential interest in using participant data 
(1) as an indication of whether appropriate individuals are being 
served, and (2)as an interpretive tool in assessing and comparing 
reported project out comes, a broader range of participant descriptors 
needs to be developed. The candidate measures can be divided into 
several categories: 

• deBOgraphic data that are easy to measure and record 
(e.g., age, sex, ethnicity); 

• descriptors of the participant's disabling condltilon and 
functioning levels (which are less easy to measure and 
categorize) ; 

• the individual's situation immediately prior to entering 
the supported employment project; and 

• previous work history and previous service history. 



different types of participants? 
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Participant Demographics 

Supported employaient programs are all currently collecting the 
basic demographic information on participants served. An issue which 
needs additional discussion in efforts to further a coneistent reporting 
system is "who is considered a participant in supported employment" — 
all individuals referred to the system, all individuals for whom an 
assessment is completed, all individuals who enter a trtiining phase of 
services, or all individuals who are currently being compensated for 
work under the program? While variations in program models may make it 
impossible to arrive at complete standardization on this issue, more 
discussions are necessary to promote the greatest possible consistency. 

Descriptions of Disabling Condition and Functioning Level 

There are two distinct reasons for collect in g data in this 
category. One reason is descriptive : to capture the variations in 
types of disabilities and functioning levels exhibited by project par- 
ticipants across projects and across states. Another reason is 
evaluative : to assess the "severity" or potential difficulty of the 
participants selected for services, both to monitor whether the group 
selected for services matches the stated program goals, and to assess 
the level of project outcomes, taking into account the characteristics 
of the clients served. For the second purpose, it may be useful to 
construct a "client difficulty index" based on the answers to the indi- 
vidual data items listed here. However, until projects finalize the 
types of disability categories and groupings they anticipate including 
in their participant population, it may be premature to suggest how to 
construct such a measure. In addition to referring to the variables 
describing disabling condition and functioning level, a client diffi- 
culty index would probably also be based on some of the variables in the 
last category — previous work and service history. 

Situation at Program Entry 

The three variables that the study team clustered into this cate- 
gory included (1) current living arrangements, (2) public assistance 
status at enrolluent, and (3) service setting immediately prior to entry 

1 ■> / 
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into the supported employment pr^^ am. These are each data items that 
sho^. I be relatively easy to lect^ since they are based on partici- 
pa-L t-fiiuj^ at a givpn point lA.me (i.e., project entry) rather than 
recoTis of exy^erience over a longer preprogram period. The 

'cL^ . if'tic n of living arrangement and public assistance status are 
impoi\ont not only as descriptors to capture the range of variation in 
participants across projects and states^^ but are also important baseline 
measures for use in tracking individual changes brought about as a 
resul of project pa^i ticipation* 

Previoiis Work and Service History 

The variables clustered in this category can be used in three 
different ways: (1) to ident-.fy participant characteristics that may 
influence the level of outcomes a participant is likely to achieve 
(l.e., capturing some aspects of client difficulty); (2) to record more 
complete descriptions of an individual's preprogram experience that can 
be compared to the eame individual's experiences after entering 
supported employment in order to construct change measures (e.g., change 
in employmf*'"" intensity, change in earnings, and change in work 
setting); and (3) to identify more completely what groups of people are 
being reached by the supported employment demonstration (and what groups 
are not being reached) in order to assess the extent of system trans- 
formation that is occurring. 

According to the individuals involved in the national consensus 
process, those participant characteristics considered important for 
inclusion in the core data E^et (see Figure 17A) included measures from 
each of the four categories of demographics, disability informtion, 
situation at entry and prior setting. The core measures were generally 
felt to be descriptive in nature rather than measures to which programs 
should be held accountable, though consensus was not reached on this 
issue. 

There was much discussion about participant characteristics and 
concern that they not be used inappropriately in conjunction with 
outcome data to target the program towards certain types of individuals. 
Historically, under a medical model, characteristics and especially 
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Figure 17A 
PARTICIPANT CHARACTERISTICS 
Core Conseusus Data Itene and Reconimended Heasuren 



Consensus Data items 



Recommended Core Measures 



Age (date of birth, with aggre- 
gate data reported in cate- 
gories Under 18, 18~21, 22-30, 
31~A0, 41-6A, 65+ older) 



mean age 

% by age categories 



Sex 



% by sex 



Primary disability (by major 
category) 



• # and % of participants with 
each primary disability type 



IQ 



mean IQ level 



Public assistance status at time 
of program entry 

— SSI 

— SSDI, etc. 



# and % of participants re- 
ceiving public assistance 
by category 



Amount of cash grant (check state 
of art for feasibility and defini- 
tion of what to include) 



mean monthly grant 



Service setting immediately prior 

to program entry , such as : 

— special education — segregated 

— special education - integrated 

— adult ed/community college 

— sheltered workshop 

— work ac tivity center 

— day activity center 

— institution 

— other SE program 

— no day program 

— employed 

— other 



% of participants from 
immediately prior service 
settings by category 

% of participants from 
day activity programs 



// of participants ever in each 
prior setting (if research shows 
this has predictive value) 



// and % of participants 
ever in each prior 
service setting 
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functional assessments have three uses, descriptive, prescriptive, and 
predictive, with the predictive end considered to be of the "highest 
good.'* The goal of this information for the supported employment 
program, however, is not one of atcempting to predict success (because 
the program is designed to succeed for those who would otherwise fail), 
but rather one of providing aprropriate accommodations to individual 
needs and providing opportunity for choice. Thus, evaluation using 
participant data would look at what works for differert individuals, not 
whether it works. It can also be used to predict costs. 

The following data items were considered core by some seminar 
particpants and supplementary by others. These issues were not resolved 
in the large group consensus process and are included as supplementary 
items in Figure 17B: 

• ethnicity; 

• primary language; 

• presence of disability by category (more than one may be 
indicated) ; 

• severity of disability (other than mental retardation), 
with the suggestion of developing a summary score or 
index for severity of disability across all types of 
disabilities ; 

• living situation (at time of program entry) ; and 

• communication skills. 

SYSTEMS CHANGE 

According t o those who are act ive in support ed employment efforts, 
pr o j e c t— 1 ev el endeavors need to be accompanied by system-level 
procedural and policy changes to enable program and individual level 
outcomes to occur within a statewide framework of coordination, support, 
and cooperation. The ability of existing systems to facilitate the 
transition to supported employment from the current configuration of 
vocational, pre-voca tional, and day activity service programs will 
depend on the ability to confront and redefine policies, procedures, 
funding streams, and program structures. Without these changes on all 
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Figure 17B 
PARTICIPANT CHARACTERISTICS 
Additional or Supplementary Data Iteas and Other Possible Measures 



Additional or Suppl enentary 
Data ItocB 



• Ethr.icity 

— White" 

— Black 

— Hiepanic 

— A£i^n 

— Native american 

— Other 



Other PoBcible MeatJures 



: > G I M . 1 c i t >■ c o I o o r i e ^ 



• Primar>* language 



a 2 with English £ic a eecond language 



Presence of disability by category 
(more than one category may be 
indicated) 



• Detailed diagnosis 



• and Z of participants with each disability type 



• X with secondary disability 



Expressive communication skills 

— unclear speech 

— clear speech 

— vocalization only 

— uses aids or sign language 



• list and number by diagnosis 



Receptive communication skills 

— understands gestures 

— understands words and phrases 

— understands conversations 



• % by category of expressive communication skills 



Assessment of severity of mental 
re tarda t ion 

— profound 

— severe 

— moderate 

— mild 

— not applicable 



• 2 by category of receptive communications skills 



Asstssmenc of severity of other types of 
disabilities (measures to be developed) 



• Z by category of severity of mental retardation 

• 2 by category of severity of other disability 
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Bystem levels, supported employment may flounder and fail to develop a 
comprehensive resource alloccition and coordination strategy. 

There is some disagreement about whether system transformation is a 
secondary outcome that results fiom effectively addressing participants' 
supported employment needs, an interim set of procedural steps necessary 
to accomplish suppor":ed employment outcomes at more than a token level, 
or whether it is a primary objective in and of itself. Soma program 
administrators, operators, and state policy makers have indicated that 
their primary concern is foi participant outcomes, and that if these 
outcomes are pursued, systems change will occur to the extent that it is 
required. Others suggest that measurement of participant outcomes alone 
is insufficient for assessing performance, noting that it may be 
possible to achieve supported employment outcomes for a limited number 
of individuals without bringing about the broader structural changes. 
These changes in the service system would be needed to ensure the 
availability of supported employment opportunities to the larger 
population that could benefit from access to these opportunities. 

If a performance measurement Bystem is to include an assessment of 
the extent to which system transformation has actually occurred, then 
system change measures rsed to be developed and implemented. A review 
of existing measurement systems reveals little in the way of existing 
measures of system change currently in use, presumably because most of 
the supported employment activity occurring to date has been in the form 
of specific projects rather than as a system transformation activity. 
However, a review of existing literature and programs has been helpful 
in identifying the areas in which measurement is needed. 

These measurement areas include (1) the extent of local systems 
changes, such as changes in local case management and referral 
mechanisms; (2) the influence of the supported employment initiative on 
the state service delivery system, including achievement of ctate objec- 
tives, and changes in service delivery mechanisms; (3) the volume of 
supported employment outcones; and (4) the funding flows or amounts of 
funding by source and the relations between the funding flow and the 
participant flow into supported employment. 
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Those systems change outcomes considered important to include in 
the core data set included the more concrete measures such as the 
proportion of individuals being served and the flow of funding by 
different agencies. Tne systems change measures included are 
constructed using data items from other clusters, as well as using an 
additional data item asking for amount of supported employment fundin^^ 
by source (see Figure 18A). 

Rather than focusing on individual core data items for this domain, 
the participants in the National Consensus Seminar chose emphasized 
summary performance measures of systems change. It was noted that when 
considering systems change, one must answer the question: 
What are we trying to change ? The suggested answers to this question 
included : 

(1) increasing opportunities for individuals with severe 
disabilities; 

(2) reducing the numbers of individuals in day activity programs; 
and 

(3) shifting funding patterns from traditional day programs into 
support ed empl oyment • 

Thus, the three measures in the core set provide the mechanisms for 
assessing the extent to which these changes have occurred. 

Measures of achievement of state objectives, shifts in local 
service delivery systems and data items requiring information data from 
outside of the supported employment system (e.g., number of participants 
in day activity programs) were included in the supplementary set of 
measures (Figure 18B). These supplementary measures, then, include 
measures and data items that are particular to states and local 
communities, requiring development and collection at state and/or local 
discretion* Constructing measures using data collected from outside of 
the supported employment system would require coordination between 
different agencies for some states — a measure of interagency 
cooperation itself. 

System change measures reflect the information that a state or 
local system will choose to collect for monitoring its own transfor- 
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Figure 18A 
SYSTEMS CHANGE 
Core Consensus Data Itei^ and Recovimended He&sures 



r ' 1 

Consensus Data Items 


Recommended Core Measures 


[The first two recommended core 
measures use data items from the 
Employment Outcomes section, in 
conjunction with data from out- 

system on number of total par- 
ticipants in other programs and 
waiting lists] 

• Amount of funding for supported 
employment programming by source 


• Volume of supported employ- 
ment outcomes over time 

• Proportion of el igible target 
D ODuT a t" T on H^nno s^TV^d 

Vi/ Li/ U ^ ^* L» .X. Vi/ A A Lp/ ^ .X. * A mL D V V ^ U 

• Ratio of supported employment 
participants to total publicly- 
funded day program participants 
(including day activity, work 
activity, and sheltered work- 
shop programs) 

• Amount and proportion of 
funding for supported employ- 
ment programEiing by different 
funding sources over time 
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Figure 1 8B 
STSTEiiS CHANGE 

Additioiuil or Supplementary Data Iteu and Other Possible Heafiures 



] Additional or Supplementary- Dato Ircps 



Other Possible Measure c 



• State objectives for systeES change 



• Evidence of state objectives of systeics change in legif 
la t ion, regulation* agency roles and relationships, 
funding cechanisms, etc. 



Achievement of state objectiveii of 
syetejns change 



Evidence o^ achievement of sy s t ems change obj ec t ives 



Changes in local service delivery 
sy s t em , e.g., 

— referral structures 

— interagency cooperation 

— central ized coordination 
responsibil i ty 

— funding sources for ongoiiig 
Fuppor t 

— involvement of parents, consumers, 
and caregivers 

— case management procedures 



Evidence of change crver time in each of the components 
of the 1 ocal se rv ice del ivery sy s t ems 



Changes in state service delivery 
mechanisms such as: 

— case management procedures 

— rate setting structures 

— ability to provide ongoing funding 

— referral mechanisms 

— documentation and monitoring 

— interagency coordination mechanisms 



Evidence of change over time in each of the components of 
the state service del ivery system 



of participants in supported employ- 
ment (uses data from client character- 
istics) 

of participants in day activity pro- 
grams, work activity programs, and 
sheltered workshop progams (uses data 
from outside of the SE system) 



Ratio of supported employment participants to partici- 
pants in each of the following programs (who have nc u 
achieved supported employment) : 

— day activity 

— work activity 

— sheltered workshops 

Changes in above rati os over time 



Immediately prior setting (see partici- 
pant characteristics data) 



Changes in proportion of participants from different 
immediately prior settings over time 



Amount of system funding for each 
component of supported employment (e.g. 
ongoing support, administration, staff 
training) by source 



Prcrortion of funding for each component of supported 
employment which is paid for by the different funding 
sources over time 
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matioru These data would likely be collected annually and would provide 
both a statistical and descriptive record of what has occurred since the 
program's inception. This information will become increasingly valuable 
as the program's longevity increases. Not only will system change 
measures document the system's commitment to and follow- through on long- 
term plans for increasing opportunities for persons with severe 
disabilities, these measures will also prove useful for looking across 
states at a national picture of the program's accomplishments. 

SUPPORTED KMPLOYMKNT CX)STS 

The implementation of any new program or service approach raises 
questions about how much the program costs and how its costs compare to 
those of other programs. This is especially a concern when, as in the 
case of the supported employment initiative, the goal is one of system 
transformation, not merely one of starting new programs. Thus, 
observers not only want to know how much the program costs, and how 
these costs compare to other programs, but also vant to explore the 
relationship between the costs and the outcomes achieved through the 
program. 

Program costs are a critical data element in order to (1) compute 
cost effectiveness measures for the delivery of supported employment 
services; (2) track how the public investment in supported employment 
programming is changing over time; and (3) assess how the public 
and societal financial investment in supported employment compares to 
the taxp'yer financial benefits generated by the program. 

Attention to the performance of supported employment on cost 
measures implies a series of comparisons. One such comparison is 
between supported employment and other day programs, to address 
questions such as the following: 

• How do supported employment costs per participant compare 
to the costs of alternative day programs, such as day 
activity centers, work activity centers, and sheltered 
workshops? 

llo 

o 
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• How do the financial returns of supported employment (tax 
revenues and reduced transfer payments from the 
taxpayer*s perspective, and increased disposable income 
from the participant's perspective) compare to the 
financial returns generated by public investment in 
alternate day programs? 

• Do the cost savings generated justify the public outlay 
for these programs? 

A second set of comparisons is among different supported employment 
proj ectB, and particularly across different models of service delivery 
(e,g,» across projects that utilize individualized job settings versus 
mobile work crew models versus enclaves within industry) to answer 
questions such as the following: 

• How do the costs of supported employment vary from 
project to project? 

0 Do cos t— ef f ec t iv enes s or benefit— cost measurers vary 
according to the model of supported employment that is 
being used? 

It is important tci note here the need for caution in conducting 
comparisons such as these. Care will need to be taken to ensure that 
program context is taken into account as well as the characteristics of 
program participant s. There are many variables that can influence costs 
besides the operation of the programs themselves. 

In negotiating consensus about the cost data items that should be 
considered core or universal data elements, several constraints were 
identified. First, not all supported employment projects monitor costs 
in a way that would permit them to identify the actual costs associated 
with serving a given individual. (Group-oriented models, for example, 
are less likely than individual placement models to keep track of the 
specific levels of support provided to individual participants). Thus, 
the recommended core measures focus on collecting aggregate project data 
on program costs and computing mean costs per participant, rather than 
on recording costs for each participant. Second, a number of proj ects 
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would find it difficult or impossible to distinguish the costs 
associated with different program activities (e.g., ongoing support 
versus supervision, since in many projects these activities are closely 
intei-twined or conceptually merged). Thus, the recommended core data 
items ask only for total program expenditures. 

Figure 19A summarizes the data items that are recommended for 
universal dpta collection, and the recommended perforLnance measures that 
are based on these dava items. The core data elements document 
supported eaplpyment funding sources and expenditure levels for a given 
reporting period (such as an annual period), and document any 
restrictions on the services, participants or time frames imposed by the 
funding sources. P'.ablic funding as well as privately— generated revenue 
are included. 

The recommended performance measures based on these data items 
summarize total project expenditures from public and other revenues, and 
compute nean total annua ] cost per enrollee. nean annual public cost per 
enrollee, and the mean public cost of supporting one participant in 
supported employment for one montlu A final recommended performance 
measure utilizes aggregate statistics on mean monthly participant 
earnings (described under Em ployment Measures ) to compute the mean 
public cost per dollar of participant earnings. 

Figure 19B summarizes a series of supplementary cost analyses that 
are needed to answer performance questions about the costs, benefits, 
and relative cost effectiveness of supported employment compared to 
other programs. Each of the items listed in Figure 19B really 
represents an extended set of data elements and a series of 
computations. 

The first supplementary analysis involves comparing the total costs 
and mean monthly costs computed for supported employment projects to 
data on the costs associated vith other day programs. For the purposes 
of comparison with other programs, the full costs of supported 
employment also include the cost of participation in other day services 
as an adjunct to supported employment. In addit a to comparing total 
program costs, it will also be important to do comparisons using 
standardized units such as "cost per participant hour" and "cost per 
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figure 19A 



SUPPORTED EMPLOYMENT COSTS 
Core Consensus Data Items and Recommended Measures 



Core Consensus Data Items 



Recommended Measures 



Asount of Public Fundinp. by Source 

• Funding agency 

• Total amount by funding source 

• Constraints by funding source 

— Funding constraints about time 
frame (s ) 

Funding constraints about service 

— Funding constraints about eligible 
or target populations 



and type of publ i c funding sources 
Amount of pub lie funds from each source 
Total public expenditures during reporting period 
% of pub lie funds from each source 



Amount of Other FundinR by Source 

• Nonpublic sources of funding 

— Foundation grants 

— Community fund raising 

— User fees 

— Revenue generated by sale of 
•products 

• Toti: I amount by funding source 

• Constraints by funding source 

Funding constraints about time 
frame (s) 

Funding constraints about services 

— Funding constraints about eligible 
or target populations 



# and type of other revenue sources 
Amount of funds f rom each source 

Total expenditures of other-than-public funds during 
rep or ting period 

% of other revenues f rom each source 



Total Program Costs 

• Total public costs for supported 

employment during (annual) reporting 
periods 



• Total expenditures (public and other) 

• % of total expenditures that are public 

• Mean total cost per enrol lee 

• Mean public cost per enrol lee 

• Mean public cost per person month of supported 
empl oyment 

• Mean public cost per dollar of participant earnings 
during reporting period 



Total Public and Private Costs DurinR 
(Annual) Reporting period 

(For revenue producing projects and 
projects with nonpublic sources of 
support for total expenditures including 
revenue generated by project or other 
non-public funds) 

• Total enrollees served during reporting 
period* 

• Total individual person-months of 
supported employment accumulated 
during the reporting period* 

• Total earnings by supported employment 
participants during the reporting period 



*See Data Items under Employment Outcomes 
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Figure 19B 
SUPPORTED E^tPLOYK^NT COSTS 
Additional or SuppXeaentary Data Items and Other Possible Measures 



BEST COPY AVAILABLF 



Additional or Supplementary 
Data Items 



T 



Relative Costs of Alternative, Adjunct or 
Complementary ProRraxs Utilized by Supported 
Employment Participant s 

• Total conthly cost of other prograns during 
reporting pence 

• Total monthly cost per enrollee in paid work 
through other programs 

• TotaJ monthly cost per participant dollar earned 
through other programs 

• Cost of participation in day services as an 
adjvnct to s up p or ted em pi oyment 



Other Poss i bl c Meae ur es 



Taxpayer Impacts; Computation of Taxpayer Benefi t 
Costs for Supported ETiploynent Programs 

• Mean monthly transfer payments 

• Mean monthly public cost of social services 
(including cost of day programs) 

• Mean value of tax revenues generated 



Societal Impacts; Computation of Societal 
Benetit~Cost for Supported Elmployment Programs 

• Mean value of participants' contribution to 
the social product (estimated using earnings) 

• Noneconomic benefits such a& changes in 
quality of life experienced by participants 
and other family members 

• Mean monthly total cost of social services 
received by participants 



Gene rot 1 cn of summa ry cost daia; comparison cf cor.t 
mecLSures (^ee recummenQeG measures ) jetweer. 
oupporr ed em pi oyment and other day programs such 
shtr*-_-r- i work activity programs, day activity 

program, r 



Ge.- erati on of summary cost data; comparison of 
taxpayer costs and benefits between si-.r'orted 
employment and other programs controlling for 
participant charact erist ics and other factors 



Generation of summary cost data; comparison of 
societal costs and benefits berween supported 
empl oyment and other programs control ling for 
participant characteristics and other factors 



Participant Impacts; Computation of Participant 
Benefit Costs for Supported Employment 

• Mean monthly earnings, net witheld taxes 
e Mean reimbursed work expenses 

• Mean monthly transfer payments 



Generati on of summary cost data: compa rison of 
mean participant f»xpendable income berween 
supported employment and other programs 
controlling for participant characteristics 
and other fact or s 



Employer Impacts; Costs and Benefits of Employing 
Supported Versus-'Non^Supported Workers 

• Provision of training and supervision 

• Job accommodation costs 

• Productivity levels 

• Absenteeism 

• Job turnover rates 

• Wages paid 

o Fringe benefits provided 



• Generation of summary cost data: mean net costs 
(benefits) of emp.cyin^ supported workers (ir. 
comparison to labor industry standards or 
co-workers in similar jobs) 



Costs by Type of Project Activity 

^ percentage of total budget 

- cated to general administration. 
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Percentage of budget allocated to each type of 
activity over time 
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participant dollar earned." The generation of data on costs of other 
programs would have to be undertaken at the state level or local system 
level, since these data are not available to local supported employment 
proj ect s. 

The second supplementary analysis involves comparing the financial 
costs and benefits of supported employment from the perspective of the 
taxpayer. To generate these measures, it will be necessary to collect 
da t a on : 

• supported employment program costs; and 

• other changes in public costs resulting from participa- 
tion in supported employment, such as: 

— changes in transfer payments received by participants 

— changes in participant utilization of othei publicly- 
funded community services 

— s av ing s from increased t ax r ev enu e s paid by 
participants. 

It is likely that collection of these data, even for a limited research 
sample, will be beyond the capacity of most supported employment 
projects, unless special research funding and research staff are made 
available. In order to compare the taxpayer benefits and costs from 
supported employment to the other day programs, similar data on costs 
and benefits would have to be generated for these other programs as 
well. 

The third supplementary analysis, involves comparing the costs and 
benefits of supported employment fromthe perspective of society as a 
whole. To generate these measures it will be necessary to collect data 
on : 

• total costs of support ed employment and other social 
services to participants including non-governmental costs; 
and 

• the value of participants* production or contribution to 
the productivity of society as a whole — i.e. net 
contribution to the social product. 
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This analysis also provides the specific vehicle for addressing the non-- 
economic benefits of the program such as increased community integration 
and participation and improved quality of life for participants and 
their families. 

The fourth supplementary analysis involves comparing the financial 
costs and benefits of supported employment from the perspective of the 
program participant. In computing these measures, it will be necessary 
to collect data on: 

• unreimbursed participant expenses associated with par- 
ticipating in supported employment; 

• increases in participant take— home earnings; 

• changes in earnings of other household and family members 
iue to reduced need for participant care; 

• changes in transfer payments or other financial support 
received by participants; and 

• estimated value of any changes in life quality 
experienced by participants. 

Like the previous taxpayer benefit/cost measures, the collection of data 
for the participant benefit/cost measures is likely to be beyond the 
capacity of individual supported employment projects, even for a limited 
study sample, except as part of a specially— funded research effort. 

The fifth supplementary analysis listed on Figure 21B is the 
collection of data on the benefits and costs to employers hiring 
supported employees. Once again, a variety of data would have to be 
collected for a sample of participating employers, including: 

• additional training and supervision costs; 

• other job accommodation costs; 

• costs or cost savings from supported worker productivity 
compared to other workers; 

• costs or cost savings from supported worker absentee- 
ism/job turnover rates compared to other workers; and 

• costs or cost savings from supported worker wages and 



fringe benefit-costs compared to other workers. 
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These data are not likely to be part of the ongoing data collection 
efforts of a supported employment project. A special research study at 
the state or federal level could be used to generate data on these 
measures • 

The final supplementary analysis which may be of particular 
interest to some projects or some project models is an analysis of how 
total project costs are allocated to various activities, such as general 
adniinietration, outreach, job development^ direct participant super- 
vision and/or training, and indirect case management services. The 
measures based on these data would be used to comp^.re the percentage of 
the project budget allocated to different activities, as well as shifts 
in these percentages from one reporting period to another, or from one 
project to another. 
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V. DEVELOPING A DATA COLLECTION STRATEGY AND IMPLEMENTATION FLAK 

CONSISTENCY VERSUS VARIATION 

By suggesting cctLtLor. data elements and measures that may be used 
across different supported employment projects and/or states, the set of 
core consensus measures represents an attempt to define common 
approaches to measurement rather than mandatory or standardized informa- 
tion systems. Variation or flexibility in data collection practices 
across projects and across states will be expected due to: 1) 
variations in project models, 2) how long the projects have been in 
existence, 3) particular local or state characteristics, and 4)the 
priority given to different objectives. For example, on a local level, 
the data collection and monitoring system for a new free— standing pro- 
ject which uses group work stations in an urban industrial area with 
accessible public transportation and a priority on serving individuals 
from day activity centers may look somewhat different from an informa- 
tion system for a project using an individual placement model based in 
an existing service organization located in a rural area. 

On a state level, data systems will also vaxry based on whether an 
existing information system is expanded to include supported employment 
variables or whether an entirely new system is developed. The 
utilization of an existing system may provide a state with a broader 
base of participants, but may be more limited in the types of data able 
to be collected, and the timing of that collection. Definitional 
problems may also arise when expanding an existing state system, given 
any changes from historically accepted to current terminology. Other 
states may choose to design and implement an entirely new data system 
especially fur supported employment programs. While this may enable 
those states to develop a unique system reflecting the timing, 
definitions and participants involved particularly with supported 
employment, a separate system may be limited by its specific supported 
employment focus rather than on providing more global information about 
the entire potential target population. Thus, a range of both local and 
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state level data systems have and will continue to emerge along with the 
development and refinement of supported employment programs. 

However, there does appear to be keen interest in potential for 
promoting consistency in data collection where possible, through the 
defining and iniplementation of a set of broadly applicable measures such 
as the ones presented in Chapter IV. These may be useful to projects 
for internal uses, including the refinement of service practices and the 
self-monitoring of project development. Additional potential uses of 
standardized data include accountability to funding agencies and "cross- 
fertilization" through sharing of information between programs which are 
collecting data using the same data elements. The suggested core data 
set also represents a framework for state systems to consider in 
designing monitoring systems for grantees, in order to include data on 
outcomes the states want to encourage as well as to include measures 
they want all projects to be held accountable for. On a national level, 
these measures represent the building blocks for federally initiated 
evaluation or research about supported employment, and prcrvide a frame- 
work for designing such an effort which addresses the necessity for 
basic uniformity in definitions, data elements, and documentation 
procedures. 

Whether or not a large-scale federal overview of supported employ- 
ment efforts ever occurs, this study's findings are an encouragement for 
states and projects to determine the important questions to be asked 
about the effectiveness of supported employment efforts and the best 
ways to answer key performance questions. The core or minimum set of 
measures described in this report are measures that are intended to be 
used flexibly by states, with respect to individual service providers. 
Some states may only be able to look toward implementation for programs 
receiving specific supported employment funding. Others will be able to 
collect information from all programs offering supported employment 
alternatives regardless of funding source* However states choose to 
implement a system of program monitoring and evaluation, it will be 
critical to carefully document: (1) the range of participants served; 
(2) the accomplishments of the program; (3) the best practices; and 
(4) systems changes with an eye toward generating a rich data base of 
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common ini '^rma t ion on a c ross— pr oj ec t scale as well as the unique 
aspects of particular programs. 

OVERVIEW OF IMPLEMENTATION OPTIONS 

Who Collects the Data and Who Maintains a Client-level Data Base ? 

With the exception of systems change and costs measures, the core 
and supplemental data items and measures described in the previous 
chapter are generally stewed in the form of client-level data elements. 
However, state or national level core consensus measures could be con- 
structed from project-level summary reporting measures, rather than 
necessarily requiring the maintenance of a client-level data base at the 
state or national level. 

There are two basic alternatives for collecting client level data 
— either at the project level or at the state level. In either case, 
it is the service provider maintaining the participant files who will 
have ultimate responsibility for initially extracting and recording 
these data* Thus, there are two basic options: 

• The data from the individual client files are reproduced 
and simply passed on to the state for aggregations; or 

• Projects maintain the only set of par t i c ipan t- 1 ev el 
records due to concerns about: 



part icipant confident iality , 
familiarity with the information, 
consistency of the data, and/or 

choice of state to not collect or require reporting 
of client-level data? 



In this last instance, projects would prepare aggregate statistics for 
submission to state-level monitors/evaluators. 

Who Aggregates the Data and Generates Summary Reporting Measures ? 



describing the project, state and national perspectives on supported 
employment accomplishments. As mentioned previously, a client-level 



Figure 20 illustrates options for collecting and reporting data 
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Figure 20 
Data Collection and Reporting Options 



Client-Level Measures 



Option A ; Project collects 
data on clients 



Option B; Project collects 
data on c ients 



Project enters 
client data into 
project-level MIS 

_^ Project enters 
client data into 
project-level MIS 
for its own use 



Project generates 

project-level 

statistics 

Project also pro- 
vides state with 
copy of the client- 
level data base 



Project reports 
statistics to 

iitate 

State uses this 
client-level data 
base to generate 
summary statistics 
for project 



Option C : Project collects 
data on clients 



Project provides 
state with paper 
records on each 
client 



State enters data 
into a state-wide 
client-level data 
base 



Project Level Measures 



Project collects 
information on 
project-level 
measures (e.g., 
costs) 



Project provides 
state with this 
information 



State uses this 
client-level data 
base to generate 
summary statistics 
for project 



System-Level Measures 

State or local system administrator State receives and coordinates 

\l I collects information on system-level ^ system-level measures 

measures (e.g., total volume of 
participants participating in supported 
employment) 
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data base can be maintained at either the state or the project levels. 
Which way this occurs and how often the data are summarized depends upon 
state and local project negotiations and intended uses of the data. It 
may be more meaningful for the projects' internal use if the summary 
statistics are prepared and reviewed by project staff before the data 
are passed on to the state. If aggregation and preparation of summary' 
performance measures occurs at the state level, however, states can be 
assured of consistency and quality control across projects and can also 
somewhat relieve the burden of data management at the project level. 
Again, who does what and at what level needs to be negotiated among the 
involved actors- System level measures, as shown in Figure 20, will 
need to be generated and analyzed at the state or federal level. 

Timing Issues 

The timing of data collection and reporting activities is another 
important aspect to consider. It seems likely that most projects would 
prepare quarterly reports of the collected information. The timing of 
the preparation of data summaries would vary somewhat, depending on the 
nature of existing information systems and whether client-level or 
project-level data are being used. Thus, reporting measures may be 
collected on an ongoing or monthly basis even though they are likely to 
be reported only quarterly, or, in the case of cost and system change 
measures, annually. 

Some measures may be collected at program entry (for baseline data) 
for each participant and again when changes occur in participant status. 
Collection of data as changes occur would require an ability to manage 
data collection in terms of client time. That is, the ability to trigger 
a data collection activity at any time during the reporting period. In 
some projects, rather than initiating data entries when changes occur, 
information on current client status may be collected at the end of each 
reporting period. 

Who Analyzes the Data ? 

All involved actors in supported employment program efforts will be 
interested in analyzing the data, or portions of the data, for different 
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purposes. Beyond information for internal monitoring and external 
accountability, projects will find the data useful for disseminating 
state-of-the-art information to service providers across projects and 
across different states through conferences, publications, and direct 
sharing of experiences. 

States will analyze information to: 

• look at accountability issues; 
9 de\'elop funding criteria: 

• identify barriers and disincentives for employment ; 

• devel op and refine policies; and 

• design and fund training and technical assistance 
activities. 

On a periodic or voluntary basis, national or cross-state analyses 
would provide information on variation and commonalities among projects 
and states, as well as overall changes in the service delivery system. 
Nationally, some combination of data from other sources, such as labor 
market characteristics, would also be useful as a part of this national 
effort. Finally, at all levels of aggregation, consumers, parents^ and 
caregivers will be looking toward this information to assist in informed 
decision-making about services, increased options, and improved quality 
of life fcr participants. 

PERSPECTIVES ON PROJECT AND STATE ACCOUNTABILITY 

During the initial stages of implementing a supported employment 
data collection system, most projects will be in the start-up phases of 
project implementation. Even the experienced projects will still be 
adjusting to the new mechanics of defining performance data items, and 
collecting and summarizing data on project performance measures. Under 
these circumstances, it is recommended that the initial outcome data 
generated by a supported employment information system be used as 
descriptive information about program experience, and as broad 
indications of whether state projects and the national program initia- 
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tive are moving in the desired directions, but not as measures of the 
relative effectiveness of different projects or program models. 

Ultimately, after the initial start-up phase of data collection is 
completed, performance measures can become useful for assessing indi- 
vidual project accomplishments as well as national performance. The 
potential assessment-related uses of the proposed data items and per- 
formance measures include (1) ongoing self-assessment by projects them- 
selves; (2) ongoing monitoring of project performan-:e by an external 
funding agency or administrative agency; and (3) summary measures of 
system change and system effectiveness. 

Performance assessment efforts for supported employment will be 
most effective in furthering program objectives if all relevant actors 
agree on which measures should be used to demonstrate program 
accomplishments 11 as on the relative priority among different 

measures of perfor ..:e. If this level of consensus can be achieved and 
maintained, then any performance assessment exercise will be viewed as a 
way to further agreed-upon program goals. 

Performance assessment becomes dange::ous only when the performance 
measures are viewed merely as an externally imposed monitoring tool that 
can be used to reward or sanction projects based on how well they 
perform. The danger in such a situation is that a supported employment 
project might decide to "play it safe," i.e., enroll only participants 
who they know will succeed in supported employment jobs. This would be 
counter to the program goals since the whole concept of supported 
employment is to take a chance on serving individuals who have already 
been rejected as bad risks by all previous employment service providers. 
If accepted as reflecting consensus about program objectives, it is 
hoped that performance measures can have the opposite effect — that of 
encouraging projects to enroll individuals who have not previously had 
the opportunity tow ork, and of disseminating information about hov to 
enable such individuals to succeed in supported employment. 
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REFINING AND USING THE PERFORMANCE MEASURES 
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VI. USING THE PERFORMANCE MEASURES 
TO ANALYZE PROGRAM EFFECTIVENESS 

In this chapter, we briefly outline the rypes of series of qiialitative 
and quantitative analyses that can be conducted using the core anc 
supplementary performance measures described in the previous chapters. 
Each of these analyses is likely to be conducted at various scales of 
operation and levels of aggregation, depending on the variations in the 
questions asked, the resources available for data collection and analy- 
sis, and the organizational priorities of those funding the research 
efforts. In this chapter, we describe two distinct uses of performance 
measurement: (1) to conduct ongoing analyses to inform program 
management decisions, and (2) to carry out research efforts to guide 
policy and program development. 

The basic building block required for most of these analyses will 
be the collection and recording of basic client-level data by supported 
employment projects around the country on (1) the participants, (2) the 
services provided, and (3) the ou*:comes experienced by supported employ- 
ment clients. Additionally, project-level data on program costs and 
service models will be critical to most analysis efforts. Thus, it is 
our hope that the collection of a core set of client-level data on an 
ongoing basis will come to be viewed as a part of the normal cost of 
"doing business" as a supported employment service provider. However^ 
because collection does have clear costs associated with it, the data 
designated as "core data" clearly have to be keep to a minimum. 

The purposes of BPA's study have been two-fold: (1) to facilitate 
discussion and encourage the development of consensus about the most 
important core data items and performance measures which, it is hoped, 
will come to be considered the bare minimum for data collection and 
reporting for the National Supported Employment initiative as a whole 
and for its constituent state and local projects, and (2) to establish a 
conceptual f ramew ork for the collect ion and analysis of additional data 
to address supplementary performance questions of interest to a wide 
audience. Some of these additional or "supplementary" data elements may 
be feasible and appropriate for data collection by some or most projects 
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on an ongoing basis for all supported employment participants. Others 
may be feasible or appropriate for data collection by some projects on a 



sample of all participants (i.e., a special study sample). The 
collection of still other data elements is clearly beyond the resources 
and/or capacity of local projects, and would require special research 
funding and/or special data collection efforts. 

The remainder of this chapter presents a conceptual overvie^^ of the 
various analyses of supported employment data that would address ongoing 
program management concerns and support pol icy and program development 
efforts. Later chapters describe an effort to arrive at consensus about 
research priorities for supported employment in the immediate future. 
These research priorities consist of topics which require special study 
efforts and about which a variety of audiences have expressed interest. 

DATA ANALYSES TO INFORM PROGRAM MANAGEMENT DECISIONS 

Figure 21 summarizes the different analyses that could be conducted 
to inform program management decisions by suppor ed employment operators 
and administrators. These management-related analyses of program 
effectiveness have several features in common: (1) they are based on 
directly observable measures of immediate program outcomes; (2) they are 
based on data collected and reported by projects themselves on a regular 
basis; (3) they compare the observed performance to a formal or informal 
performance standard based on previous experience or some a priori 
expectation or goal; and (4) they are intended to identify flaws or 
weaknesses in performance in a timely fashion, in order to stimulate 
creative thinking about program refinement or modification. 

At the local project level, analysis of this type is oriented 
towards "self-monitoring" and self-corrective actions. As described in 
Figure 21, the first step in project self-monitoring is describing the 
various features of the program including the services provided, the 
cost of those services, the number and characteristics of the partici- 
pants served, and the current service outcomes for each parti c ipan t 
served. During a second step, the observed data are compared to formal 
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Bjaocfll^Projcct! 
ScH-Monitori nj^f |ort8 



?f?P,A_:_Dc5Criptive Analysis 

• Describe Prof^rfliD Servicen 
and Procedures 

• Describe Prop.rim Costs 

• Describe PatticipantG 

• Describe Participant Outcomes 



_Step Bi Assessment of Effectiveneos 



• CoBipare performance to: 

— stated goals 
previous performance 

- experience of other projects 



Step C! Program/Operational 
Refinement 



• Refine progren procedures (e.g., 
participant recruitoent, design of 
service package, employer targeting, 
follow-on cervices) 

• Refine prograo goals 

• Consider alternate service nodela 

• Refine staff development end training 
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B^_r^iccUtiiDin_iatjf^^^^ 
Pjoj ec t Honit or^i ngj|f f or " 



I Describe ProgrflQ Services 
and Procedures 

• Deficribe Prof,n^m Cents? 

• Describe Participants 

• Describe Local Context 

• Describe Participant Outcomes 

• Describe Cost-Effectiveness 
Outcomes 



Ste p B; Assesc men t of Effectiveness 



• Compare performance to: 
— stated i^oals 

experience of other projects, 
controlling for variations in 
participants, labor markets 

" experience of other projects, 
controlling for variations in 
project design 



Step C: Refinement of Priorities/Development 
Technical Assistance Strategy 



• Develop and refine funding gcals 
a:,J priorities 

I Develop performance expectation? for 
"beat practices^^ or outcomes 

• Develop technical asaistance resources 



ByJy^t^cm^Adminif^t rater: 
andj^oiiimunijj/ {:\m\^v 



Jl-LtLj!^^ ipt i ve_^Ana 1 y i; ii 

• Desiribe Supported Knirl Mymmt 
Rcfiources ami Activitn-: 

• Describe Systeir Linkii^y"; 

• Describe Jlhifts in Service 
Utilization ^attern« 



Patterns 
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• Describe Shifts in Comtnnnity 
Attitudes 



Step B; Ao sessttient of ['ilfci livencsi; 



• Compare Community end Systms 
Outcomes to: 
- previous patterns 
stated goals 



_Step C: RcfipeJiriorUi^^ 

• Develop procedural linkaprs strategy 

• Develop funding strategy 
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or informal expectations about what the project status should be, and 
any discrepancies noted. Sometimes observed shifts in performance are 
merely noted to see whether a new trend is beginning to emerge. At 



may suggest that a problem is arising that requires immediate corrective 
action (e.g., if the number of job terminations initiated by clients 
increases dramatically during a given reporting period^ or if there is a 
sharp reduction in the availability of health benefits to supported 
employment participant s). 

The possible areas of project refinement in response to self- 
monitoring are far-reaching, ranging from a refinement of employer 
recruitment practices or an expansion of follow-on services available 
after placement, to a refinement of program goals or an enrichment of 
staff training. In an extreme case, the findings from self-monitoring 
efforts could cause a complete shift in choice of a service model (e.g., 
targeting a different type of industry for supported employment place- 
ments) • 

When used by an administrator responsible for monitoring a number 
of different supported employment projects, the analysis of performance 
data takes on a more comparative and explanat ory emphasis. Thus, a 
state monitor, for example, would be interested not only in how the 
experiences of different supported employment projects vary, but in the 
extent to which participant characteristics, labor market contexts, and 
service models account for variations in observed outcomes. 

As shown in Figure 21, one result of an analysis of the effective- 
ness of different supported employment projects by a state monitor might 
be to refine the state funding priorities for future funding periods. 
Another outcome might be to develop criteria describing 'T^est practices" 
or minimally acceptable performance levels to be used in monitoring 
local projects. A third response might ^e to develc»p a technical assis- 
tance curriculum for local projects to disseminate '^best practices." 

The third level of data analysis shown on Figure 21 is analysis by 
the administrator most closely responsible for the system transformation 
and community change objectives of the supported employment initiative. 
Initially, at least, it seems clear than the data analysis associated 



other times, however, a performance measure observed by project managers 




ERIC 



123 

with monitoring systemB transformation objectives will be descriptive in 
nature* Shifts in service utilization patterns and resource expenditure 
patterns will be compared to previous utilization patterns and to state 
goals. Based on these comparisons, strategies for continued systems 
change may be developed or refined. 

DATA iWALYSIS TO SUPPORT POLICY AND PRCX;RAM DEVELOPMENT 



Data analysis to support policy and program development efforts 
have the following features in common: (1) they are attempts to go 
beyond the immediately observable short-term program outcomes to assess 
longer-"term net program effects in light of stated program objectives; 
(2) they are likely to be conducted on a special project basis, rather 
than as part of an ongoing reporting effort; (3) they are usually 
referred to as research or evaluation efforts, rather than as reporting 
or monitoring efforts; (4) they may or may not be able to utilize data 
collected on an ongoing basis by program operators; (5) the findings are 
usually of interest to a larger audience of observers than just program 
operators and administrators in assessing the success of previous 
policies and determining whether to continue or amend the tested program 
designs. 

Although the distinction between program monitoring efforts and 
program evaluation is not usually as clear as it has been presented 
here, it is nevertheless useful to consider the distinct types of analy-^ 
sis called for by the two efforts. The analyses that are part of a 
research and evaluation-oriented assessment of cupported employment are 
needed to conclusively answer questions about whether the public invest-^ 
ment in supported employment is causing measurable progress towards 
fulfilling the objectives of the program, and what the relative balance 
of program costs and program benefits is, from both the taxpayer and 
participant perspectives. Research analyses are also intended to clari- 
fy what program designs and practices are most effective in achieving 
desired outcomes for what types of participants:; that is, to identify 
how to replicate successful outcomes in other sites (and perhaps also 
how to avoid the unnecessary repetition of unsuccessful outcomes). 
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Figure 22 BummarizeB several of the analytic tacks likely to be 
included in research analyses of supported employment. These tasks have 
been divided into three phases: 

Phase I: An Explanatory Analysis of Immediate Supported Employ^ 
ment Outcomes; 

Phase II: Tracking and Analyzing Longer- Term Participant Outcomes 

and Net Impacts; and 
Phase III: Comparing Net Program Benefits and Costs. 

The first phase — conducting an explanatory analysis of immediate 
supported employment outcomes — consists of many of the same activities 
described under performance monitoring in the previous section. The 
difference here is that the analyses are oriented towards developing 
summary measures of effectiveness and cost-effectiveness for the 
supported employment program as a whole, and xa identifying the factors 
that influence the observed outcomes, rather than towards assessing the 
performance of any particular project. 

In the second phase — tracking and analyzing the longer-term 
outcomes and net impacts of supported employment — the research 
analyses begin to diverge farther from the performance monitoring 
analyses. All supported employment projects will theoretically continue 
to collect data on a participant so long as he or she is in supported 
employment as part of the the core data recommended for regular 
collection and monthly or quarterly reporting. However, regular 
reporting measures are not well designed to track a given individual or 
cohort of participants after entry into the program. An important set 
of research analyses will be to track participant status over time in 
terms of earnings levels, employment stability, employer satisfaction, 
participant job satisfaction, and other quality of life indicators. 

In addition to longer-^term longitudinal studies of a given cohort 
of supported employment participants, the research agenda for assessing 
supported employment impacts also includes tracking the experiences of a 
carefully selected control or comparison group not participating in 
supported employment, in order to determine how supported employment 
participants would have fared had they not entered supported employment. 
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Analyse s to Support jolic}|jndJro3ra£peyelo£Dien^ 



Phase I: Analysis of Process 
and Ifflinediate Outcomes 



9 Describing seniceE, costs, 
participants, participant 
outcoies, systems change 
outcomes of the SE program 
es a vhole 

I Comparing imniediate outcomes 
to stated goals 

• Analyzing factors influencing 
outcomes 

— participant characteristics 

— labor market environments 

— service strategy 

• Developing effectiveness and 
cost-effectiveness measures 



Phase II: Tracking and 

Analyzing Longer-Tenn 

Outcomes an d Net Impac ts 

8 Conducting longitudinal 
studies to track impacts over 
time of supported employment 
on: 

employment stability 

— earnings available 

— total income 

— cost-effectiveness measures 

I Using comparison or control 
groups to identify net 
program impacts on participants 

• Conducting further explora- 

ti,iir; of program impacts on 
lifo juality 



Phase III: Comparing M 
Program Benefitt; to Net 
Program Cost6_ 

J Identifying all fiscal 
impacts of the program from 
the taxpayer's perspective 

• Identifying all fiscal 
impacts of the program from 
the participant's perspective 

• Identifying both economic 
and nonecononiic impact of the 
program from society's 
perspective 

I Comparing the value of the 
stream of supported employ- 
ment costs and benefits from 
the taxpayer participant 
and societal perspectives 
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The difference between the mean observed outcomes for the supported 
employment participants and the comparison group constitutes the mean 
"net" impact that can be attributed to supported employment. 

Finally, additional research is needed on how to measure several of 
the supported employment program objectives that are not easily 
quantifiable, such as increased social integration and improved quality 
of life. While these objectives reference strongly held beliefs about 
the benefits of supported employment for program participants, they have 
not yet been successfully translated into validated measures for use on 
an ongoing basis in or as part of special research studies. 

The third phase of research analyses — comparing net program costs 
to pet benefits from both the taxpayer and the participant perspect ive 
is a methodological tool for summarizing all the program impacts that 
can be quantified in dollar terms and comparing the total balance sheet 
of costs and benefits from the perspective of the taxpayer contributing 
public funds to support an individual in supported employment, and from 
the perspective of the program participant. The findings from benefit/ 
cost analysis can be variously expressed (1) as the ratio of the current 
value of program costs (incurred now and to be incurred in the future) 
to the current value of program benefits (experienced now and to be 
incurred in the future), or (2) as the number of months or years until a 
financial 'T>reak-even" point is reached. What is important to keep in 
mind in this brief sketch of benefit cost analysis is that the costs 
included in benefit/cost analysis include not only the direct increased 
costs of supported employment programming, but also any increases in 
other public costs associated with supp or t ed employment participation 
(e.g., costs resulting from increased utilization of community support 
services). Likewise, the benefits, from the taxpayer perspective, 
include not only any reductions in the cost of day service programming 
utilized, but also indirect benefits such as increased tax revenues paid 
by the supported employment participant and reductions in the level of 
pub lie assistance support received.^ Benefit/cost analysis is 
controversial because of its emphasis on assigning a dollar value to 
each desired outcomes. However, it does offer a way to provide 
information about whether the supported employment program makes sense 
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as a financial investment, from a purely economic point of view. A more 
comprehensive approach to benefit/cor analysis would include analysis 
of non-economic benefits also. 

The following chapters discuss the priorities for the research 
agenda for supported employment in more detail, and describe the need 
for the collection of additional data to support these research efforts. 
Two of the types of research analyses described above are most likely to 
require data collection sbovc and beyond the efforts undert.aken by 
ind-^.vidual projects. The first is the analysis of data on outcomes 
experienced by comparison or control group members (since these 
individuals are, by definition, not included in a project's client-level 
data base). The second type of analysis that would require additional 
data collection is the analysis of net costs and benefits, since this 
analysis requires drawing on so many additional data bases for data 
(e.g., on tax revenues paid and on the cost of services provided by 
other community agencies to supported employment participants. 
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NOTES 



^Of course, from the participant perspective, any reduction in 
transfer income would be viewed as a cost, not a benefit. 
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VII. IDENTIFYING RESEARCH ACTIVITIES 
OVERVIEW OF THE ROLE OF RESEARCH IN ASSESSING PROGRAM PERFORMANCE 

Part II of this report has outlined a core set of consensus 
measures that could be implemented in a standardized way across projects 
and/or states to summarize major aspects of program performance. 
Additional measures have also been presented that an individual project 
or state might want to include in its ongoing data collection and 
analysis for use in program management. These recommended core measures 
and proposed supplementary measures have been developed with careful 
attention to the feasibility and appropriateness of implementation on an 
ongoing basis and are built upon the availability of existing measures 
in the field. This is the first such effort to identify common 
performance measures for supported employment programs and is occurring 
at a time when nationwide efforts are still in their infancy. It is to 
be expected, then, that there are gaps in the existing array of measures 
where further research is needed. Similarly, there are areas of 
performance measurement where ongoing collection and analysis of data 
for the universe of participants would simply be too burdensome or where 
a one-time research effort is more appropriate. 

There are five major types of evaluation research to consider when 
building a research agenda for assessing the performance of supported 
employment programs and conducting analyses to support policy and 
program development : 

(1) development of new measures where none currently exist 
or where the current state-^of- the-art is inadequate to 
fully address a given measurement domain; 

(2) collection and analysis of data for a representative 
sample of participants over time and/or in conjunction 
with a comparison group; 

(3) collection and analysis of data from outside of the 
supported employment program such as the amounts of 
transfer payments received or taxes paid, parallel data 
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collection for a comparison group, or information about 
relevant environmental fact ors ; 

(4) observation of program performance by an outside 
observer such as in the development of qualitative 
studies or ident if icati on of "best practices" ; and 

(5) secondary analysis of existing data using complex 
statistical analyses or mathematical modeling such as 
in the multivariate analysis of factors contributing to 
success or in the development of bene f it-cost models . 

The following pages review each of the performance measurement 
domains described earlier and identify research activities that» in 
conjunction with the core and supplementary measures described in Part 
II» provide the basis for a comprehensive evaluation of supported 
employment efforts. The final chapter prioritizes a research agenda 
with recommendations for those activities for which there is the most 
apparent and immediate need. 

IDENTIFYING RESEARCH ACTIVITIES FOR EACH OF TriE MEASUREMENT DOMAINS 

As described in Part I, the process of developing proposed measures 
to be considered during the consensus— building process involved identi- 
fying a wide range of program objectives and performance questions to be 
addressed in assessing program performance. As described in Part II of 
this report, many of these aspects of performance can be appropriately 
addressed through ongoing performance monitoring. However, within each 
of the eight identified measurement domains there are a number of 
aspects of measuring program performance that have been identified as 
being more appropriate as one-time or periodic research and evaluation 
act ivit ies. 

EMPLOYMENT OUTCOMES 

As described in Chapter IV, ongoing data collection and project- 
level reporting offer the potential for monitoring a range of employment 
outcomes including measures of employment status, earnings, hours 
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worked, and length of time employed. However, these data are not 
sufficient to address the full range of performance questions about 
employment outcomes identified through the consensus process. Thus, the 
following areas of additional research have been identified as essential 
to a comprehensive assessment of program performance: 

• changes in participant earnings and employment stability 
over time; 

• comparison of participant earnings with those of 
traditional habilitation and day activity programs; and 

• comparison of participant earnings and employment 
stability with coworkers or employees in comparable jobs. 

Changes Over Time 

While the supplementary measures proposed in Figure 12B include 
employment status and mean earnings at various points in time following 
enrollment, they are summarized across the total program and do not 
include a comparison of specific cohorts of participants at different 
points in time. Thus, the proposed supplementary measures include the 
percentage of participants employed and participant: mean earnings dui. Ing 
the fourth, eighth, and tw el f th q uarters f ol low ing enrol Im ent. Data 
being collected on an ongoing basis for individual program participants 
could be further analyzed to provide a more comprehensive picture of 
changes in earnings and employment stability over time. We propose that 
such analyses would be conducted for individuals in a selected sample of 
participants and might include such measures as: 

• mean weeks worked during second two quarters of program 
participation as a percentage of weeks in the first two 
quarters; 

• mean weeks worked during the f.econd (or third) year as a 
percentage of weeks worked in the first year; 

• mean monthly earnings during second two quarters of 
program participation as a percentage of earnings in the 
first two quarters; 
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• mean monthly earnings during the second (or third) year 

of program participation as a percentage of earnings in 

the first year; and 
9 liieBtj hours w orked per week during fourth, eighth^ or 

twelfth quarter as a percentage of hours worked in the 

second quarter after enrol Icncint . 

Assuming that the sample is representative of the participants in a 
given program (if large) or across a state, it will be possible to use 
these outcome measures in further analyses that explore factors asso- 
ciated with "success." That is, over time, as larger numbers of par- 
ticipants reach their second and third year following enrollment and as 
program models become more established, it may be possible to identify 
specific approaches that contribute to long-term successful outcomes. 

Comparison with Other Programs 

The comparison of earnings of supported employment participants 
with those of participants of other programs is considered by mo£t 
concerned individuals to be a key aspect of measuring performance. This 
is especially true in light of systems change strategies that emphasize 
shifting resources from existing habilitation and day activity programs 
to supported employment. 

This comparison will require collecting data from a comparison 
group of traditional habilitation and day activity programs or from 
summary statistics on the outcomes of these other programs and 
analyzing these data in conjunction with a representative sample of 
supported employment participants. It will be important to match 
comparison groups in terms of participant characteristics or control for 
participant characteristics when conducting the analyses. This kind of 
analysis would probably be conducted as the supported employment program 
matures. However, it is not the kind of analysis that would be needed 
on an ongoing basis or that would need to be conducted for every project. 

The following are examples of the kinds of measures to be included 
in an analysis of this kind: 
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• mean hourly wage during hours worked of supported 
employment participants as a percentage of dean hourly 
wage of participants in: 

— sheltered workshops, 

— work act ivity programs, 

— other vocational programs targeting individuals with 
severe disabilities, and 

— day activity programs, 

who are not involved in supported employment; 

• mean month ly earnings of supported employment 
participants as a percentage of mean monthly earnings of 
participants of : 

— sheltered workshops, 

— work activity programs, 

— other vocational programs targeting individuals with 
severe disabilitiep and 

— day activity programs, 

who are not involved in supported employment. 
Comparison with Coworkers 

One of the central concepts of supported employment is the 
"normalization principle," which simply maintains that individuals with 
disabilities be given the opportunity for experiences that are equiva-- 
lent to those of the general population to the maximum extent possible. 
Thus, when assessing outcomes such as duration of employment and 
earnings levels, it is important to view them in the context in which 
the individual participants are working. 

This comparison is included here as a research issue rather than 
being included in the ongoing measi.res in Chapter IV because it lequires 
collecting additional data from outside the supported employment pro- 
gram. It is anticipated that these data would be collected for a 
selected comparison group and compared to a selected research sample of 
supported employment participants. Measures used might include: 

• mean hourly supported employment wage as a percentage of 
mean hourly wage for comparable jobs at entry; 
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• mean month ly supported employment earnings as a 
percentage of mean monthly earnings for comparable jobs 
at entry; and 

• length of time (number of months) in same supported 
employment job as a percentage of number of months in 
same job for comparable employees. 

QUALITY OF KMPLOYMKNT 

Perhaps the most difficult aspects of performance to measure are 
those that involve the concept of "quality". Much has been written 
about the measurement of quality in human service programs. This body 
of literature has struggled to identify the aspects of quality that are 
highly valued in the field and to develop proxy measures that best 
reflect the most difficult aspects to measure. Measures of the quality 
of employment for supported employment participants are no exception. 
While a number of proxy measures can be identified, the true dimensions 
of quality are much more difficult to address. The core and sup- 
plementar^' measures described in Chapter IV include a set of employment 
characteristics that offer descriptive information about employment 
placements and the reasons why participants leave their jobs (Figures 
13A and 13B). The descriptive information provided by these measures 
can be interpreted as proxy measures of quality to the extent that some 
value has been placed by a particular program on the types of work or 
reasons for termination that are considered most desirable. However, 
the measures themselves are truly value free and though they offer some 
information about the nature of the employment situation, to effectively 
address the important aspects of the quality of participants' employ- 
ment, further development of measures will be needed. 

In a field that emphasized "normalization" it is perhaps not 
surprising that the aspects of quality of employment considered by 
respondents to be important to supported employment are the same as 
those valued for anyone in the workforce. The following areas have been 
identified as important to include in a research agenda for assessing 
the quality of employment for supported employment participants: 



EKLC 



135 



m goodness of fit or appropriateness of the match between 
the job and participant's skills and interests; 

• quality of the work environment; 

• employment mobility (horizontal and vertical); and 

e analysis of the relationship between the type and quality 
of employment and other outcomes. 

Job Match/Goodness of Fit 

Job matching efforts as part of the initial placement process can 
range from efforts to provide the employer with the most highly quali- 
fied applicant through a simple matching to ensure that essential 
skills are present, to efforts to ensure that jobs offer participants 
the richest possible employment experience including stimulation, chal- 
lenge and an opportunity to learn new skills. Existing job matchinc^ 
procedures focus on collecting information about skills, behaviors, and 
job requirements. However, there are other aspects of the match between 
workers and jobs that could be further exploited in order to develop 
measures of the goodnefss of fit. These job match measures would be an 
approach to assessing the quality of participants' employment in addi- 
tion to using them during the placement process. Some examples include: 

• compatibility between how job performance is measured 
(e*g,speed of production, attention to detail, ability to 
do a wide variety of tasks) and participants' abilities; 

• extent to which the nature of the work is compatible with 
the interests of the participant; and 

• compatibility of supervisor's management style with par— 
ticipant's communication abilities and need for struc- 
ture; and 

• extent of participant's satisfaction with the job (as a 
post-placement measure rather than one for use during the 
matching process). 

There may, of course, be many other measures that are important to 
assessing the goodness of fit. As part of an evaluation effort, the 
effectiveness of various job match measures as predictors of job success 
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can be tested by comparing the scores for participants who retained a 
job for a given period with those who t.2rminated from their jobs (while 
controlling for other factors). The ^^tility of job match measures for 
ongoing performance assessment include testing whether the project or 
state has achieved a reasonable match for a high proportion of its 
participants. Another use of job match as a performance measure may be 
to decide that the job mat' hing process itself is a desirable feature of 
project operations, and then to assess whether a given project nas 
adequate procedures for identifying the job requirements and other 
relevant features of the employment placement and matching these to 
participants' characteristics. 

Quality of the Work Environment 

Discussions with consumers highlighted the importance of 
considering the work environment, as well as the nature of the work 
itself, when assessing the quality of an employment situation. (In 
fact, evidence suggests that from the consumer perspective, supervisor- 
coworker relationships, extent of emphasis on speed, and nature of the 
work environment may outweigh wages and hours as criteria for assessing 
the desirability of a given employment situation). However, before the 
quality of the work environment is measured, resear h is needed to 
determine the aspects of the work environment that are highly valued by 
participants and that appear as determinants of job success. Work is 
also needed in defining data items and performance measures that can be 
used to assess the quality of these aspects across varied work environ- 
ments. Some of the areas identified, so far, as possible aspects of the 
work environment for which measures might be identified or developed 
include: 

• the physical work environment - e.g., space, light, air; 

• the quality of interactions with supervisor and coworkers 
- e.g. , communication, information sharing, attitudes of 
supervisor and coworkers: and 

• the appropriateness of job accommodations made. 
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Empl oym en t Mob il i t y 

One of the aspects of employment quality that we value highly in 
our society is the upward mobility potential of a job. This value may 
seem to be somewhat misplaced in the context of supported employment, 
given that the target population is individuals whose potential for 
competitive employment is limited by their disability and who are 
expected to alvs'ays need ongoing support in order to sustain employment. 
However, the need for flexibility in employment options and the oppor- 
tunities for horizontal as well as vertical mobility can be as important 
for individuals with severe disabilities as for their nondisabled 
coworkers. Perhaps one of the reasons that this issue has been raised 
in the context of supported employment is because many consider it an 
employment option that attempts to improve the limited employment op-- 
tions that have been available through more traditional habilitation 
programs. 

It is important to note that the issue of employment mobility and 
the availability of a range of employment options and choices for 
participants is not valued to the same extent by all operators And 
policy makers involved in supported employment. Some consider the major 
goal to be obtaining employment options for participants. The concept 
of offering a range of options with opportunities for advancement or 
horizontal mobility seems far-fetched and a concern that would be better 
addressed many years down the road when the first basic employment needs 
are met. However, others have stressed the importance of assessing not 
only the opportunities for mobility but the actual extent to which 
participants move vertically or horizontally duriu^ their years of 
employment. Vi\rther research is needed about the importance of mobility 
to long-term program success and the ways in which it might reliably and 
easily be measured. 

Relationship of the Quality of Employment to Other Outco mes 
The concept of "quality" implies a set of values about the nature 
of the employment placement that theoretically should be closely linked 
to the length and stability of that employment. That is, a well matched 
"meaningful" job in a good environment is theoretically one in which the 
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participant is likely to remain the longest. Quality of the employment 
may relate to other outcomes also such as the impact of supported 
employment on quality of life. The analysis of these relationships can 
help guide program operators in identifying those aspects of employ aaent 
quality that appear to be most important to emphasize vhen identifying 
potential jobs for their participants. 

OPPORTONITIBS FOR WORKSITE INTEGRATION 

As described in Chapter IV, the issues of integration at the 
worksite include both the extent to which disabled individuals work in 
environments in which nondisabled coworkers are present and the nature 
and extent of interaction between disabled workers and their coworkers. 
Several measures of the degree to which nondisabled workers are present 
were identified in Figures lAA and lAB. However, a number of research 
topics have been identified that are important to assessing the extent 
of integration or elimination of segregation in the workplace such as: 

• exploration of def initional issues in measuring the pro- 
portion of individuals in a given work setting who are 
disabled; 

• development of measures of the nature and extent of 
interaction between supported employment participants and 
their nonsupported cow orkers; and 

• analysis of the relationship between worksite integration 
and other outcomes. 

Proportion of Disabled Workers 

The proxy measures of the extent of integration, included as core 
consensus measures, are limited to measures of the type of ^supervisory 
structure, whether or not nondisabled workers are present, and the 
numbers of supported workers in a group placement. Efforts to measure 
the actual proportion of disabled workers in a work setting were 
considered supplementary partially because of concerns about the 
difficulty of capturing this measure in all settings. In small 
companies, clearly defined work settings, or for certain types of 
supported employment models, the computation of a percentage of disabled 
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workers is straightforward. In those particular projects this measure 
could easily be collected on an ongoing basis. However, a large variety 
of exceptions were noted where additional work is needed to explore the 
implications of different approaches to operationalizing measures of 
proportion in areas such as: 

— the individuals to be counted on the "disabled" side 
of the ratio — only supported workers or all workers 
with disabilities? 

— what constitutes a "work setting" - how large is it? a 
single room within a larger building? the whole 
building? only the area where workers interact with 
each other? 

— how should mobile work groups be handled? how might 
their varied and changing work environments be taken 
into account? 



Extent and Nature of Interaction 

Respondents throughout the study have stressed the importance of 
going beyond the perfunctory placement of severely disabled individuals 
into environments where nondisabled people are presenti, emphasizing the 
importance of meaningful contact between coworkers as an essential 
indicator of integration. Some preliminary measures of interaction have 
been suggested, though much more work is needed to expand, test, and 
refine them before any could be broadly implemented. Examples include: 

• % of supported workers interacting with nondisabled co- 
workers : 

— during work tasks; 

— during breaks or travel to and from work; 

— not at all; 

• % of supported workers in each of the following situa- 
tions : 

— exposed to nondisabled during the work day but with 
limited interaction; 
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has Ci atinuing opportunity for interaction but work 
taeV are largely independent of the need for coworker 
interaction; 

has significant number of daily experiences which 
require cooperative interaction with nondisabled co- 
workers; 

has interactions with nondisabled coworkers to the 
same degree that nondisabled workers do; and 
develops relationships with nondisabled coworkers that 
extend beyond the work setting. 



In-depth discussion about the issues of measuring the degree of 
integration at the worksite have raised a number of important concerns 
that need to be taken into account in approaching this somewhat 
sensitive and emotionally charged issue. For example: 

• Many study respondents have cautioned against focusing 
exclusively on integration at the worksite itself, and 
have encouraged exploring the degree of integration into 
the community at large also, (See Quality of Life 
measures later in this chapter and in Chapter IV.) Other 
actors feel that supported employment programs have more 
of a secondary effect on community integration for which 
they should not be held directly accountable and thus 
consider it inappropriate as an area to include in per- 
formance measure^^. 

• One of the values held dear by many individuals involved 
in supported employment efforts is that of maximizing 
B el f - d i r e c t i on and choice for participants. Thus 
researchers are cautioned against a priori assumptions 
about the social behaviors that indicate "success". Some 
participants may not choose to socially interact to the 
same degree as others, and some participants may choose 
not to develop "meaningful relationships" with nondis- 
abled coworkers. Integration is probably something that 
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one allows to happen rather than something one causes to 
occur. 

• The degree to which a work environment provides oppor- 
tunities of integration involves not only the minting of 
disabled and nondisabled workers and the extent to which 
interaction is required or encouraged, but also a whole 
set of attitudes and changes in perceptions about the 
potential of individuals with severe disabilities as 
productive members of society. Thus, measuring the atti- 
tudes of the nondisabled supervi s or s and cow orker s m ay be 
an important aspect of assessing the extent to which 
supported employment programs are successful at reducing 
the segregation of disabled individuals in the work 
force. 

Relationship Between Worksite Integration and Other Outcomes 
As mentioned previously, one of the important potential effects of 
work site integration to be measured is its influence on community 
integration. For many participants interaction with nondisabled 
coworkers provides an opportunity to develop social and communication 
skills that strengthen their ability to function in the community. 
Conversely, for many participants success at work will be dependent on 
participation and success in nonwork environments. 

Degree of integration can be anticipated to be related to other 
outcomes also. Assuming that further research is able to yield valid 
and reliable quantitative measures of the degree of integration, multi- 
variate statistical techniques can be used to analyze the effect of work 
site integration on outcomes such as job skills, job satisfaction and 
employment stability. 

ONGOING SUPPORT 

The core and supplemental measures of ongoing support described in 
Chapter IV provide basic information about the type and intensity of 
support provided. These measures are fairly straightforward for ongoing 
data collection and can be used to conduct further research into the 
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nature and ef f ect ivenesB of ongoing support. Four areas have been 
identified for consideration when developing a research agenda: 



• further study of the nature and role of nonpublicly 
funded support; 

• analysis of the changes in intensity of support received 
over timei and 

• analysis of the effectiveness of different types and 
levels of ongoing support. 

Intensity of Support 

Currently, supplemental measures of ongoing support have been 
identified that estimate the average number of hours and costs of 
support per participant for different types of support. Discussions 
with researchers and practitioners suggest that these measures will be 
essentially project-level descriptors of the total amount of each type 
of support provided by a project, averaged over all of the project's 
participants. Further exploration of approaches to measuring intensity 
of support for different types of program models may yield practical 
solutions to the difficulties of measuring support at the client level. 
Another pos-sible refinement to be explored would be for projects to 
report individual amounts of support received by participants in 
individual placements, and group averages for participants in group work 
stations or work crews, with any aggregation of these data being done 
separately for individual and group models. 

Nonpublicly Funded Support 

In the strictest sense of the term^ supported employment implies 
support that's publicly- funded . In fact,, if this distinction were not 
made, then the differences between supported employment and regular 
competitive employment become more difficult to ascertain. However, 
even within the context of supported employment programs providing 
publicly—funded support, a variety of additional sources may also be 
brought to bear on the full range of support needs of the participants. 
Probably the most common sources of nonpublicly funded support that 
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programs are likely to encounter include the employer and the 
participant's family, charitable giving and perhaps even the purchase of 
support by the participant. 

Possible measures of nonpublicly funded support might include: 

• types of nonpublicly funded support utilized; 

• mean hours of nonpublicly funded support by source; 

• mean cost of nonpublicly funded support by source; and 

• extent of nonpublicly funded support as a proportion of 
total support: provided. 

These measures would be based on data collected for a selected research 
sample. While ongoing data collection and analysis would not be 
necessary, measuremenv. at several points in time over the next five or 
ten years would provide; valuable insights into how the relative share of 
costs shifts as the program matures. 

Changes in Intensity of Support Over Tijne 

While basic measures of intensity of support as described earlier 
may be constructed as project level measures, an important aspect of 
understanding the intensity of support provided involves exploring the 
extent to which an individual's experience changes over time. Some 
program models are specifically designed to "fade" or decrease the 
amount of support provided over time. Others may tend to decrease 
support, given the decreased needs of some participants, while main- 
taining fairly constant levels for others. In still other cases, there 
may be occasional increases and decreases over time as participants 
learn new tasks, change jobs, and/or face various life changes outside 
of work that increase their need for on the job support. In any program 
model there is likely to be some period in the very beginning of a 
participcints involvement with the program when support is particularly 
intense. 

Research into the ways support changes over time would be conducted 
for a sample of individuals employed continuously over a specific 
research period. Such a research activity might include measures such 
as : 
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• mean hours of support during the second (third, fourth, 
etc,,) quarter of program participation compared to the 
first ; 

• mean cost (assuming that costs include other expenses 
besides staff hours) of support during the second 
(third, fourth, etc.,) quarter of program participation 
compared to the first; and 

• comparison of the above for different types of programs 
and different types of participants. 

Effectiveness of Dif f erent Types and Levels of Support 
Once measures of intensity of support have been refined, and data 
have been collect <?^d over a long enough period, multivariate statistical 
techniques can be used to analyze the effect of different types and 
intensities of ongoing support on different outcomes while controlling 
for variations in program, participant, and environmental characteris- 
tics. 

QUALITY OF LIFE 

By far one of the most difficult domains to capture through 
performance measures, and perhaps one of the most controversial issues 
has bean the effect of supported employment programs on participants' 
quality of life. As mentioned in earlier chapters, some controversy 
exists about the eppropriat eness of holding programs accountable for 
their effectiveness in improving quality of life* Others propose that 
quality of life measures should simply be considered descriptive. Some 
proponents of accountability insist that quality of life is a (or "the") 
niajor goal of supported employment without which other outcomes are 
essentially meaningless. Opponents, on the other hand, maintain that 
the goal of supported employment is simply employment and that impacts 
on the quality of life are secondary effects for which programs should 
not be held accountable. 

Another controversy about quality of life measurement exists in how 
it should be measured. Some actors feel that participants' self-report 
would be the most valiu approach assuming that methods are found that 
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are effective for individuals with severe communication and cognitive 
impairments. Others recommend observational techniques or composite 
measures of multiple indicators. Some simply feel that quality of life 
is too difficult a concept to capture through standardized measures and 
will always belong in the domain of research. 

With all of the reservations and concerns that exist there is 
general consensus that the impact of supported employment on quality of 
life should be assessed. Since no measures have been identified to 
date as core consensus items, a major priority for research efforts will 
be to explore the feasibility of developing measures for standard 
implementation. Suggested areas in which research should be conducted 
include : 

• degree of community integration and participation; 

• changes in participants' expendable income; and 

• changes in other quality of life indicators. 

Community Integration and Participation 

The issues of measuring community integration are similar in some 
ways to measuring integration at the work site. The concept of 
integration combines both presence of disabled and nondisabled 
individuals in the same environment and social interactions between 
them. Another aspect of community integration that shows promise for 
inclusion in performance measures is the extent to which participants 
access other community resources and activities. Research efforts in 
the area of community integration should focus on developing measures 
that can be adopted as ongoing measures of program effectiveness. 

Expendable Income 

While the outcome measures described earlier include measures of 
earnings, increased earnings do not necessarily mean an improvement in 
financial situation. One indicator of the extent to which participation 
in supported employment improves quality of life is the effect on 
expendable income. Here the term "expendable income" is used to mean 
money available for spending on improving one's quality of life, or 
money left over after life's necessities have been paid for. It is 
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important to note that expendable income may also decrease for a given 
participant, especially if the receipt of work income affects his or 
her eligibility for public income support. 

The measurement of expendable income is particularly complex for 
this population for a variety of reasons. Perhaps one of the greatest 
causes of difficulty relates to participants' living situations. Many 
participants may live with parents who still manage their income* Thus, 
the effect on the participants* own personal income will be as reflec- 
tive of parents' willingness to trust their t^verely disabled offspring 
with their own money and choices of how to spend it as with the actual 
financial effect of supported employment on the family. Similarly, 
because of the severity of disability of the target population, many 
individuals may reside in board and care facilities or other ^iden- 
tial programs that again, may have responsibility for manaf g resi- 
dents' money or may even have sliding scale fees that are a^^fected by 
the residents' income. Some participants may change living situations 
either as a result of or in tandem with their participation in supported 
employment. New housing situations may be more or less expensive than 
previous one and may indicate an improved quality of life in and of 
themselves independent of expendable income. 

Research activities for assessing the effect of supported 
employment on expendable income would probably be conducted periodically 
for a selected research sample rather than becoming part of the ongoing 
performance monitoring process. This assessment might include measures 
such as: 

• mean increase or decrease in expendable income across 
participants; 

• dif f erences in changes in expendable incoipe for dif f erent 
types of participant s, different types of program s, or 
across different environmental characteristics: and 

• benefits or disadvantages of increases or decreases in 
expendable income. 



1 A7 

Quality of Life Indicators 

A number of indicators in addition to those mentioned earlier have 
been identified as potential measures of quality of life. Preliminary 
efforts to develop data collection instruments and a summary score or 
"index" are being pursued in several locations around the country. 
RefiT^ement and practical applications as standardized measures are still 
several years away, however. Some researchers caution against assuming 
t ransf errabil ity of measures from one community setting to another, 
where economic conditions, cultural differences and other factors may 
greatly affect the reliability of scores across locations. Examples of 
indicators being considered for further development include: 

• health status; 

• attitudes towards others; 

• self-esteem; 

• skill levels; 

• use of leisure time; 

• social and friendship networks; and 

• degree of self-direction. 

SYSTEMS CHANGE 

Chapter IV describes a variety of measures that can be collected by 
states on an ongoing basis to monitor the extent of systems change. 
There have been several additional systems change issues identified as 
important to consider in developing a research agenda that would not be 
a part of this ongoing monitoring system including: 

• effects of supported employment on the attitudes and 
hiring practices of employers; 

• relationships between funding flows and the flow of par- 
ticipants into and out of supported employment; and 

• areas in which system change seems crucial to program 
success. 

Employer Attitude^ and Hiring Practice s 

The supported employment concept challenges the most basic 
attitudes about competition and success in the American workplace* It 
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is based on the premise that all individuals should have the opportunity 
to be productive and contribute to society to the best of their ability, 
even if they are not the most qualified, fastest or most productive 
individuals in the workplace. This concept challenges employers to 
rethink how they view disabled workers, and reassess where disabled 
workers may fit into the labor force, even those workers who will never 
be independent of the need for assistance on the job. Thus, one 
important indicator of the extent to which supported employment programs 
are affecting the system as a whole is evidence that the labor market 
itself is opening up to supported workers through changes in employer 
attitudes and hiring practices. 

Relationships Between Funding Flow and Participant Flow 
One research issue that could be of great help to planners and 
policy makers would be an analysis of the effect of different approaches 
to funding supported employment programs on the flow of participants 
into and out of supported employment. While overall goals and valued 
outcomes of supported employment are of ten voiced about the intended 
shifts in the service system from traditional habilitation and day 
activity programs to supported employment or serving individuals 
traditionally excluded from vocational programs, the individuals who are 
in fact served may be greatly affected by the flow of funding into 
supported employment and the perspectives and constraints of various 
funding sources. Similarly, efforts to identify sources to fund 
specific aspects of supported employment such as training versus ongoing 
support may also affect participant flow patterns. 

System Changes Essential to Success 

Because supported employment represents such a major departure 
from traditional rehabilitation, both service providers and policy- 
makers have expressed the need for major changes in how the service 
delivery system operates in order for these efforts to be successful. 
One particular area suggesx:ed for further research is a study of the 
need for centralized outreach and assessment mechanisms. This type of 



ERIC 




149 



research would be conducted as a one-time activity using a case-study 
approach. 

PARTICIPANT GBARACTKRISTICS 

A number of research priorities were identified in the area of 
describing and analyzing the characteristics of individuals 
participating in supported employment programs including: 

• development of standardized definitions of severity of 
disability ; 

• exploring the development of a case difficulty index as 
an alternative or supplement to severity measures; 

• analyzing the variations in outcomes and service 
approaches for different types of participants; and 

• assessing the relationship of enrollee characteristics to 
proj ect or program obj ec t ives. 

Severity of Disability 

Standardized IQ scores are proposed as measures of the level of 
severity of mental retardation, although they need to be supplemented by 
other functional assessments such as communication skills. Similar 
standardized measures don't presently exist for other types of 
disabilities. Given the focus of the program on serving individuals 
with severe disabilities it was considered by most respondents to be an 
important research priority to identify standardized measures of 
severi ty . 

Case Difficulty Index 

Comparing the effectiveness of supported employment models and 
approaches across different providers and states will be extremely 
difficult without some way of taking into account the kinds of 
individuals being served by the program. Certainly characteristics such 
as severity of disability and immediately prior service setting will be 
useful, but it has b een suggested that a more comprehensive xndex of 
case difficulty be explored that might include a variety of functional 



166 



150 



and basic skills as well as characteristics such as work history and 
educational level. 

Relationship Between Part i cipant Character st ies and Outcomes 
As described earlier, there is much concern in the field about the 
use of data on participants for assessing program performance. This is 
largely related to concerns that evaluations of performance across 
different types of participants could be used to target the program" 
towards specific types of individuals. While targeting seems 
appropriate in the context of ensuring that individuals who receive 
services really need them, the intent of the supported employment effort 
would be undermined if the program were targeted towards individuals 
most likely to succeed* For most social service programs, the analysis 
of the chare ter is tics of program participants and the relationship 
between those characteristics and program outcomes is appropriately used 
to target the program towards individuals with a high likelihood of 
success. However, since supported employment is intended as an alterna- 
tive for individuals who have shown limited employment potential, this 
kind of analysis is most appropriate for assessing whether certain kinds 
of supported employment options and methods of support appear to be 
effective for different kinds of individuals, and how participants can 
be offered the widest possible range of opportunities. 

Relationship of Enrollee Characteristics to Program Objectives 
Not all supported employment programs or local projects have 
developed clearly defined objectives regarding the types of individuals 
to be targeted for supported employment services. However, global goals 
exist as part of the very definition oi supported employment. Thus, one 
component of a comprehensive evaluation effort would be to see to what 
extent the population served through this service approach is consistent 
with the program intent to serve individuals traditionally excluded from 
vocational rehabilitation* In some cases, programs or states have more 
specific objectives for populations to be targeted such as individuals 
currently in day activity centers versus those transitioning out of 
special education or specific disability groups such as the mentally 
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ill. In the cases where more specific target population objectives have 
been formulated, i comparison of participant characteristics to 
targeting objectives becomes particularly important. 

SUPPORTED EMPLOYMENT COSTS 

The core and supplementary measures described in Chapter IV include 
a range of measures of program costs and benefits that address all of 
the major cost questions that need to be addressed on an ongoing basis. 
However, there are a number of research activities that have been 
identified for inclusion in the proposed research agenda including: 

• analysis of the net fiscal impacts of supported employ- 
ment , 

• analysis of the specific costs of supported employment at 
the client level; and 

• analysis of how taxpayer costs and benefits and partici- 
pant financial benefits vary across project models; 

Net Fiscal Impacts 

The concept of net impacts is that of identifying the extent to 
which changes that occur are attributable to the intervention, or the 
extent to which outcomes exceed those that would have occurred in the 
absence of the program. The analysis of net fiscal impacts requires 
the collection of data from outside of the supported employment systems 
(e.g. transfer payments, social services utilized) own data base, and 
from a comparison group of nonparticipant s from outride of the supported 
employment program. 

An analysis of net fiscal impacts from a taxpayer perspective would 
involve collection of data for a research sample of both supported 
employment participants and a comparison group on items such as: 

• transfer payments (SSI, SSDI, other cash assistance, food 
stamps, medical benefits) received during a baseline 
period ; 

• transfer payments received during a study period after 
the participants entered supported employment; 

• cost of social services utilized during a baseline period 
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(day programs, residential programs); 



cost of social services utilized during a study period 
after the participants entered supported employment 
(including taxpayer cost of participation in supported 
employment proj ect) ; 



tax revenues paid during a baseline period; and 



• tax revenues pai^ during a study period after the 
participants entered supported employment . 

Analysis of net financial impacts on participants would include data 
such as: 

• uncompensated work expenses incurred during a baseline 
period; 

• uncompensated work expenses incurred during a study 
period after the participants entered supported 
employment ; 

• earnings, net of taxes withheld, during a baseline 
period; 

• earnings, net of taxes withhexd, during a study period 
after the participants entered supported employment; 

• transfer payments receiving during a baseline period; and 

• transfer payments received during a study period. 

Client Level Costs 

Cost measures as described in Chapter IV are based on project-level 
costs rather than costs for specific individuals. Collecting client- 
level cost data is particularly problematic for program models where 
staff time is shared across a number of individuals in a group setting 
or where support and supervision are provided simultaneously by the same 
individual. However, for a selected research sample, it would be 
feasible to construct cost measures on an individual basis that would 
allow for a finer level of detail in analyzing the costs associated with 
different approaches and levels of support. 
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Variations in Benefit/Cost Ratios Across Models 

The analysis of variations across different types of supported 
employment models would use existing data being collected for other 
measures. Comparisons of society, taxpayer^ and participant 
benefit/cost measures across different program models would be conducted 
on a periodic or ad hoc basis once the program matures and benefit/cost 
models have been refined. A comprehen;;ive analysis would also include 
noneconomic costs and benefits. 

IDENTIFYING BEST PRACTICES 

The research activities described above are each specific to a 
particular domain of measurement. However, across the whole range of 
supported employment performance questions there is an important cross- 
cutting issue that must not be overlookedL As described in Chapter III, 
there are many important evaluation questions about how 
various practices, procedures, program models, and the relative priority 
given to various program objectives can influence program outcomes. 
The evaluation of practices and procedures contributing to success might 
be conducted by an outside party and would include: 

• gathering descriptive information about program implemen- 
tation procedures and service processes; 

• gathering informed opinion about the practices that seem 
to be innovative and particularly effective; 

• conducting a cross— site analysis of information collected 
about practices and procedures in conjunction with 
outcome data; and 

• exploring the transferability and feasibility of prac- 
tices in varied settings. 
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VIII. PRIORITIZING THE RESEARCH AGENDA 

The previous chapter outlines a broad and ambitious set of research 
activities that have been identified during the course of this study as 
important to a comprehensive assessment of the performance of supported 
employiaent programs and the impact of the national supported employment 
initiative. However^ not all of these research activities are 
appropriate for immediate implementation, nor are all necessarily of 
equal importance. In order to develop a reasonable and practical 
research agenda it is important to prioritize among these various topics 
and activities. There are a number of questions to ask about these 
research issues in order to prioritize them into a workable agenda 
including : 

• How important is the timeliness of the research? How 
soon are the findings of the research needed? 

• Has the program had suf f icient experience to offer an 
adequate data base for the research? Is the program 
sufficiently mature for this research to be appropriate? 

• To what extent will findings of the research be important 
for improving program performance? Will the findings 
affect social policy? 

DEVELOPING MEASURES FOR ONGOING PERFORMANCE MONITORING 



Using the above criteria, one concludes that the first priority for 
further research will be development of measures in some of the areas 
where adequate measures and standardized definitions do not currently 
exist. Should the development of standardized measures in these areas 
prove feasible, these measures can be included in the existing core and 
supplemental data sets for ongoing collectiou. Activities to develop 
and refine measures to be included in ongoing data collection are an 
early priority so that important information ±e , c lost during these 
initial years of program operation. Priority areas for development and 
refinement of ongoing measures are as follows: 
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• measures of participant characteristics including: 

— severity of disability, and 

— overall case difficulty; 

• measures of the degree of integration including : 

— refinement of measures of the proportion of disabl^^d 
individuals in the work setting, and 

— measures of the extent and nature of interaction 



The development of measures in these three areas are also recommended as 
a first priority because of their importance to the other kinds of 
research to be conducted. Assessing net impacts and benefit/cost can 
not readily proceed until better descriptions of program participants 
are developed. The vocational rehabilitation (VR) and developmental 
services (DD) systems currently use quite different disability 
categories and definitions of the severity of disability that are not 
only incompatible with each other but aie also considered by most 
respondents to be inadequate for this program. Similary, identifying 
approaches to program improvement will be difficult without better 
descriptions of the types and intensity of support provided and the 
degree of integration offered through existing models. 

Other areas for development and refinement of ongoing measures were 
al so identified. These are areas in which research might be of a long 
term nature and while potentially equally important to the three areas 
above, should perhaps be investigated concurrently with, rather than 
prior to, identifying program improvements and assessing net impacts and 
benefit cost. These include: 

• measures of the quality of employment including : 



between disabled and nondisabled coworkers; and 
• refinement of measures of the intensity of suppor t 
provided. 



— the goodness of fit or appropriateness of the match 
between the job and participant's skills and 
interests, and 

— quality of the work environment; 
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• measures of quality of life including potentially 
developing a composite index of a variety of quality of 
life indicators if feasible; and 

• measures of the costs and benefits of supported 
employment . 

IDENTIFYING APPROACHES TO PROGRAM IMPROVEMENT 

The second major priority for the research agenda are those 
activities that can produce readily usable results for improving program 
operations. Once outcome measurement approaches have been refined, 
three major types of research activities are needed to identify the 
factors that contribute to program success: 

• collection of data on environmental characteristics from 
outside of the supported employment data system; 

• multivariate statistical analysis of the relationships 
between various program characteristics, environmental 
factors, participant characteristics and outcomes; and 

• collection and analysis of implementation processes and 
service procedures to identify "best practices". 

ASSESSING NET IMPACTS AND BENEFIT-COST 

The third major priority for the research agenda is assessing net 
impacts and summarizing relative benefits and costs of supported 
employment. While this topic may not be any less important from a 
policy point of view than the above two, it is less urgent in terms of 
the timing of the research and the need for maturity of the program 
before activities are undertaken* While the above research activities 
would ta?:e place ir; an environment where a commitment to funding and 
implemenv.ation is alr<iady in place, this third priority is essential to 
making poli'^.y derision:* about further investment in or expansion of 
supported employment opportunities for individuals with severe 
disabilities. 
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NEXT STEPS TOWARD INITIATING RESEARCH ACTIVITIES 

This study has helped lay the groundwork for the identification of 
not only the relevant and priority research topics, but also of some of 
the current "experts" in particular aspects of supported employment. It 
will be important to continue this identification process in order to 
build on the existing knowledge base through the exchange of theories 
and practices. As communication linkages hopefully increase between and 
acroc^s projects, states, research groups, and involved federal agencies, 
it will be important to encourage further discussion of the relative 
importance of research activities. This information from the field at 
large will be essential, as federal policy makers consider what research 
activities to encourage and support. Similarly research institutions 
and state agencies themselves may wish to be guided by the issues raised 
here as they develop their own research agendas. 

Perhaps one of the most important lessons to be noted is the 
importance of effectively utilizing information developed through the 
experiences of programs and researchers in similar or related fields. 
The BPA study included reviews of materials from special education, 
school-to- work ".ransition, sheltered employment, and traditional 
vocational rehabilitation programs. However, much more cross- 
fertilization is needed in order to make the best use of knowledge from 
these fields. As we look towards developing future research activities 
it w ill be important to take advantage of existing data bases, data 
collection mechanisms, measurement approaches, and instruments to ^he 
maximum extent possible. This implies the need for further dialogue 
between key researchers and policy makers in the supported employment 
field and experts in fields such as residential services and special 
education. 
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CONCLUSION 

The contents of this volume are not intended to provide a single 
fixed, final, or authoritative set of performance measures to be applied 
to all supported employment programs. Rather, the intent has been to 
provide a consistent but flexible approach to measuring performance for 
supported employsGnt. Within each project and/or state system a set of 
data items and measures can evolve that meets the particular program 
configuration, policy interest, and information system capabilities 
present. Thus these measures represent an approach that can be 
customized to the needs of each state in such a way that they can be 
translated and aggregated across states. 

As stated elsewhere in this report, the development of performance 
measures that can be used to assess program success and relative project 
effectiveness and cost— effectiveness in furthering supported employment 
objectives will be a gradual process. The first necessary phase in this 
process will be the implementation and monitoring of comparable data 
collection practices across a variety of projects and states, to ensure 
data quality and to describe the range of program design included under 
the rubric of supported employment. Only after the variations in 
program models are well understood and the comparability and accuracy of 
reported data have been ensured can the second phase of performance 
measurement begin* 

During the second phase of performance measurement, administrators, 
evaluators, and researchers will be able to begin using these data to 
address issues of relative effectiveness and cost effectiveness of 
supported employment compared to other programs, and of different models 
of supported employment compared to each other. It should be cautioned 
that using performance measures to compare supported employment projects 
to another will never be a simple activity. Comparisons across programs 
must take into account variations in the characteristics and previous 
experiences of the participants served by the programs, as well as 
variations in the labor market contexts of the project sites. Perhaps 
the most direct application of the performance measures will be for a 
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single project to use them in monitoring its own success in meeting 



The most complex levels of performance assessment — the net impact 
and benefit/cost analyses described in the previous chapter — are not 
likely to be pursued by a single project or even by a single state 
acting independently. Strong and coordinated national leadership will 
be necessary as a catalyst for progress in this area for two reasons. 
First, the level of resources that will need to be devoted to these 
analyses is substantial, and is likely to be committed only with 
considerable federal assistance. Second, the types of questions 
addressed by these analyses are likely to be federal policy issues. 
These issues are (1) whether to continue to encourage the shift of day 
program resources from day activity and work activity programs towards 
supported employment, and (2) ultimately, whether to expand the 
opportunities for supported employment by increasing the level of 
resources devoted to this program. 



desired goals or improving on its past performance levels. 
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Ruth Aledort 
Katie Banzhaf 

iJrnie Baumann 

Tom Bellamy 

Elizabeth Boggs 

Shawn Boles 

Jeannine Boyer 
Bruce Brown 
Lou Brown 

Carhy Chambliss 
Michael Collins 

Ron Conley 

John Denine 

Paul Dziedzic 
Keith Foster 
Mar>- Falvey 

Karen Flippo 

John Flanders 



Affiliation 



Job Path, New York 

Star Center, Connecticut 

Tran:*it ional Empl oyment 
Project, New Jersey 

Specialized Training xerogram. 
University of Oregon 

Researcher and Consumer 
Represent a tive 

Specialized Training Program, 
University of Oregon 

AHEDD, PA 

Dept. of Rehabilitation, CA 

Rehab il i tat ion Psychol ogy and 
Special Educa t i on. University 
of Wisconsin, Madison 

Utah VR 

Special ized Training Program, 
University of Oregon 

Admini s trat ion on Develop- 
mental Disabilities^ DHHS 

University of Washington, 
Food Service Project 

Washington VR (Blind Agency) 

Dept. of Rehabilitation, CA 

Dept. of Special Education, 
L-A. State University, CA 

Rehab il i tat ion Admini s trat ion , 
University of San Franc isco» CA 

Minnesota VR 



Type ot Respondent 
Project Level 
Project Level 
Project Levea 

System Level 

System Level 

Consul tant 

Project Level 

System Level 

System Level 
System Level 

System Level 

Project Level 

System Level 
System Level 
System Level 

Project Levex 

Systel Level 



Robert Gay lord -Ross 



Dept. of Special Education, Project Level 

San Francisco State University, CA 
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Debbie George 
Geoffrey Goedecke 
JitL Gonzalves 
Terry Hibpschman 
Janet Hill 

Mark Hill 

David Johnson 

Bob Jones 

Bill Kiernan 
Margaret Lamb 
Frank Laski 
Mark Litvin 

Pat Malzahn 
David Mank 

Ron Mannini 

Sherril Moon 

James Moss 

Bruce Nil son 
Jack Noble 



ARC, San Diego» CA 

People First, CA 

Ea s t B ay Re gi o n ai Gen t e r , CA 

Kentucky VR 

Department of Mental Health and 
Mental Retardation, Virginia; 
formerly VCU 

Rehabilitation Research and 
Training Center, Virginia 
Commonweal th University 

University Af f il iated Proorams 
University of Minnesota , 
Minneapol is 

Rehabilitation Services Admin. 
Washington, DC 

Children' s Hospital , Boston, MA 

Dept. of Rehabilitation, CA 

Researcher, NIHR Fellow 

Department of Rehab il itation , 
Col orado 

ARC, San Francisco 

Supported Employment Technical 
Assistance, Univ . of Oregon 

Vocational Training Enrichment 
Davis, CA 

Rehabil itation ResearcVi and 
Training Center, Virginia 
Commonweal th University 

University of WA; 
formerly RSA/Dept. of Ed. 

AHEDD, PA 

State University of New York 
Buffalo, NY 



Type of Respondent 
Project Level 
Advisory Committee 
System Lv-vel 
System Level 
System Level 

Advisory Committe 't 
System Level 

System Level 

Project Level 
System Level 
System Level 
System Level 

Project Level 
System Level 

System Level 

System Level 

System Level 

Project Level 
System Level 
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Candace O'Neill 
Mark Pfeffer 

Harvey Pressman 

Ian Pumpian 

Betty Rabinkoff 
Grant Revell 

Larry Rhodes 

Bill Rosenberg 

Richard Rosenberg 

Frank Rusch 

Irvin Rutman 

Wayne Sailor 

Bob Schalock 

Jim Schmidt 
Richard Schutz 

Steve Shestakovsky 
Joel Smith 



Affiliation 



O'Neill and Associates, WA 

Exceptional Children' s 
Foundation, Los Angeles, CA 

Corporation for Opportunity 
Expansion, MA 

Dept. of Special Education, 
San Diego State University, CA 

Arizona VR 

Dept . of Rehab il i tat ive 
Services, Richmond, VA 

Special ized Training Program, 
University of Oregon 

Training Resource Group, 
Special Education Resource 
Network, Hayvard, CA 

Career Assessment and Place- 
ment Center, Whittier, CA 

Dept • of Special Education, 
University of Illinois 

Matrix Research, 
Philadelphia, PA 

Dept • of Special Education, 
San Francisco State University, 

Dept. of Psychology, Hasting 
College, Nebraska 

Foundation House, New York City 

Office of Career Development 
for Special Programs, 
University of Illinois 

Bay State Skills Corp., Boston 



Morgan Memorial Goodwill 
Boston, MA 



Type of Respondent 
Advisory Committee 
Project Level 

System level 

Project Level 
and System Level 

System Level 

System Level 

Project Level 

System Level 

Project Level 

Systein Level 

Project Level 

System Level 

Consul tant 

Project Level 
Consul tant 



Proj ect Level 
and System Level 

Project Level 



CA 
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Type of Respondent 



Michael Smuli 

Dale Thomas 

r.rai^ Thornton 

Timm Vogelsberg 

Barbara Wayne 
MiLe Weber 
Paul Wehman 

Claude Whitehead 

John Wilberding 
Tom Yowell 
Steve Zivolich 



University of Maryland 



Stout Research and Training, 
University of Wisconsin, Madison 

Ma thaiB8 tics Pol icy Research^ 
Inc. , New Jersey 

D jvelopmental Disabilities 
Program, Temple Univ., PA 

Michigan VR 

East Bay Regional Center, CA 

Rehab il itation Reserch and 
Training Cent er , Virginia 
Commonwealth University 

Administrati on 

Devel opmental Disabili ties 

Washingt on, DC 

Goodw ill Indus t ries , 
Milwaukee, WI 

Washington Div, of Develop- 
mental Disabilities 

Regi onal Center of Orange 
County, CA 



Project Level 
and System Level 

System Level 



Advisory Committee 
Consul tant 

System Level 
System level 
System Level 

System Level 
Project Level 

System Level 

Project Level 
end System Level 



APPENDIX B: 
NATIONAL CONSENSUS SEMINAR PARTICIPANTS 



181 



NATIONAL SOPPORTKD EMPLOYMKNT OONSEWSDS SEMINAR PARTICIPANTS 



Ruth Aledort 

Director of Dissemination 
Job Path 

22 West 38th Street 
New York, NY 10018 
(212; 9AA-056A 

P. David Banks 
Rehabilitation Research and 

Training Center 
Virginia Commonwealth University 
131A W. Main Street 
Richmond, VA 23214 
(804) 257-1851 

Bracha Ben-2ui 

National Insurance Institute 

of Israel 

Jay Buckley 

Employment Network - Technical 

Assistance Project 
135 Education Building 
University of Oregon 
Eugene, OR 97 403 
(503) 686-5311 

Michael Collins, Ph.D. 

405 Waterman Building 

Special Education, Social Work 

and Social Services 
University of Vermont 
Burlington, VT 05405 
(802) 656-2936 

Terry Cote 

Department of Mental Retardation 

State of Connecticut 

90 Pitkin Street 

East Hartford, CT 06108 

(203) 725-3919 

Sharon A. Davis, Ph.D. 
Associaton for Retarded Citizens 
2501 Avenue J 
Arlington, TX 76006 
(817) 640-0204 



Paul Dz iedz ic 
12936 - 79th Place. NE 
Kirkland. WA 98033 
(206) 721-4422 

Theda Ellis 

Division of Vocational Rehabilitation 
Puch F. Mail Stop 0581 
State Office Building 
Juneau. AK 99811 
(907) 465-2814 

Ron Garcia 

Las Cumbres Learning Services, Inc. 

P.O. Box 653 

Los Alamos. NM 87544 

(505) 662-2812 or 662-4323 

Janet Hill 

Department of Mental Health and 

Ment al Re ta rda ti on 
P.O. Box 1797 
Richmond, VA 23 214 
(804) 786-1746 

Mark Hill, Ph.D. 
Rehabilitation Research and 
Training Center 

Virginia Commoviweal th Jniversity 
1314 W. Main Street 
Richmond, VA 23 214 
(804) 257-1851 

David R. Johnson, Ph.D. 

University of Affiliated Programs 

Room 6, Patte Hall 

150 Pillsbury Drive, S.E. 

University of Minnesota 

Minneapol is, MIC 55455 

(612) 37^-8383 

John Johnscr 
Fort Wright School 
501 Farrell Drive 
Covington, KY 41011 
(606) 331-7742 
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Susan Kidder 

Division of Vocational Rehabilitation 
131 W. Wilson Street, 7th Floor 
P.O. Box 7852 
Madison, WI 53702 
(608) 266-1281 

David Mank 

Specialized Training Program 
College of Education 
Univesity of Oregon 
Eugene, OR 97 403 
(503) 666-5311 

Richard Melia 

U.S. Department of Education 
National Institute on 

Handicapped Research 
Office of Special Eductaion and 
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APPENDIX D: 



CURRENT OSERS DEFINITION OF SUPPORTED EMPLOYMENT 

(Developed for National Demonstration Projects) 



DESIGNING SUPPORTED EMPLOYMENT DEMONSTRATION PROJECTS 



Because Supported Employment Demonstration Projects constitute a 
new priority in the Rehabilitation Services Administration's Program of 
Special Projects and Demonstrations for Providing Vocational 
Rehabilitation Services to Severely Handicapped Individuals, this 
additional information is provided to assist applicants in design of 
proposed projects. The information is descriptive, not regulatory. 
Information in the regulations takes precedence if there is any apparent 
conflict. 

The design of a supported employment demonstration project requires 
four steps: 

1 . An analysis and description of the current system of 
ongoing day and vocational service s for per sons with 
severe disabilities . This analysis should include 
descriptions of the State's administration of the program 
or programs, provide information on the characteristics 
of local services, and show exactly how many service 
recipients are currently engaged in supported employment. 
Additional information on program philosophy, waiting 
lists, current program outcomes and so on, should be 
included, as should a description of how services 
targeted for the supported employment program differ from 
other day and vocational services for persons with 
disabilities in the State. 

2. A description of the desired statewide system of support- 
ed supported employ m ent* Th is requires a detailed 
analysis of the system of services the State expects to 
have in place at the end of the proposeu project. 
Information on anticipated State administration and 
evaluation procedures, characteristics of local services, 
relationships with the business community, and consumer 
outcomes should be included. 
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3 » An analysis of the discrepancy between the current and 
desired system of services. This analysis should include 
a precise description of the critical features of service 
content, delivery, management, and administration that 
must be addressed to eliminate discrepancy. 

4. Development of jproject objectives to remove the 
identified discrepancy. These objectives and activities 
should constitute a comprehensive, longitudinal and 
coordinated effort to move systematically from the 
current to desired system of supported employment 
services. 

It is anticipated that different states will identify different 
issues to be resolved in order to implement statewide supported 
employment programs. Nevertheless, most states will be able to achieve 
lasting statewide change only by developing strategies for: adjusting 
state plans, regulations, and funding and evaluation procedures for day 
services to reflect the characteristics and outcomes of supported 
employment; developing procedures for inter— agency coordination in 
establishing and funding supported employment; expanding the work 
opportunities available to persons engaged in supported employment; 
developing the capacity of existing and new community organizations to 
provide supported employment; building a cadre of staff who are skilled 
in providing supported employment; and informing consumers, parents, 
advocates, employers and others about the nature and purpose of 
supported employment. Naturally the strategies that will be effective 
in resolving these and other issues will be determined by the 
characteristics of the State, the existing services, and employment 
opportunities. 

Both assessing the current service delivery system and projecting 
the desired one require a thorough understanding of supported employment 
and a reliable process for determining when an individual is or is not 
engaged in supported employment. The definition in the regulations for 
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this program establishes four criteria for supported employment. To be 
in supported employment^ an individual must be (1) engaged in 
employment^ (2) in regular (integrated) work settings, (3) with ongoing 
support, and (A) he or she must experience a disability so severe that 
ongoing support is essential to maintaining employment. 

To help applicants apply these criteria to evaluate their current 
services and define the statewide system that should result from the 
project, we recommend use of the following measures and standards to 
determine whether an individual or program fits the definition of 
supported employment: 

1. Employ m ent . Supported employ ment is paid employment 



frequent daily social interactions with people without 
disabilities who are not paid caregivers. Since few 
state or local agencies currently are able to describe 
the extent of integration of individuals in day services, 
we recommend that the following criteria be used to 
estimate the capacity for integration in supported 
employment: an individual's work can be considered 
integrated when he or she works in a place (a) where no 
more than eight people with disabilities work together 
and which is not immediately adjacent to another program 
serving persons with disabilities and (b) where persons 
without disabilities who are not paid caregivers are 
present in the work setting or immediate vicinity. 



which cannot exist without a regular opportunity to work. 
An individual should be considered to meet the employment 
criterion if he or she engages in paid work for at least 
an average of our hours each day, five days per week or 
another schedule offering at least 20 hours of work per 
week. This standard does not establish a minimum wage or 
productivity J evel for supported employment. 



2 . Integration . 



Work is integrated when it provides 
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For example, an individual who works in a local bank 
creating microfilm records of transactions clearly meets 
the integration criteria for supported employment. So 
do: six individuals with disabilities who work together 
in an enclave within an electronic factory; a mobile 
janitorial crew that empl oy s f iv e persons with 
disabilities in community work sites; and a small bakery 
that employs persons with and without disabilities. 

While integration is much more likely when persons with 
disabilities work singly or in small groups among persons 
who are not disabled, the social interactions necessary 
for integration are also possible in other program sizes. 

3. Ongoing support . Supported employment exists only when 
ongoing support is provided. An individual should be 
considered to be receiving ongoing support: (a) when 
public funds are available on an ongoin ' basis to an 
individual or service provider who is responsible for 
providing employment support, and (b) when these funds 
are used for interventions directly related to sustaining 
employment . 

4. Severe disability . Supported employment exists when the 
persons served require ongoing support and is 
inappropriate for persons who would be better served in 
time-limited preparation program s leading to independent 
employment. The priority for the Supported Employment 
Demor*stration Projects is those individuals: (1) who 
previously have not been served or served successfully by 
vocational rehabilitation because of the lack of ongoing 
services needed to sustain employment after time-limited 
rehabilitation se rv ic e s are completed. With the 
development of supported employment programs in a state* 
however, it is expected that the vocational 
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rehabilitation agency will provide services to these 
individuals that lead to successful closure into 
supported employment; and (2) who are or may be funded 
for ongoing services in day programs. If those 
individuals who fit these two criteria are included on a 
priority basis, a state may also use the Supported 
Employment Demonstration Proj ect s to establish supported 
employment for other groups of individuals whom it 
chooses to fund for ongoing day services. 
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